Broadcom (LSI/Avago) HBA and RAID Controller Technical Discussion

[nezach]

Broadcom (former LSI, Avago) HBAs and RAID controllers are very popular in STH, FreeNAS and other communities. In spite of their popularity there is a surprising lack of clear technical details and a lot of ambiguity about these cards. There are some good forum and blog posts delving into some of the technical aspects, but they are few and far between.

I wanted to create this thread in order to facilitate technical information sharing and deepen our understanding of these products. I am by no means an expert on these cards and I have a limited understanding of certain technical subjects so information I post may or may not be correct, I will try to post original source of information wherever possible.

 

[nezach]

History

Knowing history and origin of the technology often time helps to better understand it, so first let’s delve a bit into LSI’s history. Their history is surprisingly complicated, so I am only including relevant events here.

• 1981 - LSI Logic founded / company specializing in ASIC technology
• 1998, Jun - acquired Symbios Logic (Symbios Products 1999) / maker of SCSI HBAs and ICs
• 2000, Jun - announced Fusion-MPT architecture (press release)
• 2000, Jun - introduced SYM53C1030
• 2001, Mar - announced integrated mirroring on Fusion-MPT (press release) / IM firmware (IR predecessor) for Fusion-MPT based fibre channel FC929 controller
• 2001, Jun - commercial availability of LSI53C1030 (press release) / first SCSI Ultra320 IoC built on Fusion-MPT architecture; based on ARM 9
• 2001, Sep - acquired AMI's MegaRAID software IP + RAID division (press release)
• 2003 - released LSI53C1035 RoC / first RoC built on Fusion-MPT architecture; based on ARM 9; MegaRAID controllers and HBAs start using "same" silicone
• 2004, Jan - delivered first SAS controller SAS1064 (press release) / first generation SAS/SATA controller; based on ARM 9
• 2005, Apr - demonstrated SAS RoC SAS1078 / first SAS/SATA RoC; moving from ARM 9 to PowerPC 440
• 2006, Feb - introduced first SAS HBAs running Integrated RAID (press release) / IR firmware
• 2008, Sep - sample shipments of SAS2008, SAS2108 (press release) / second generation SAS/SATA controller, based on PowerPC 440
• 2009, Apr - acquired 3ware RAID adapter business of AMCC (press release)
• 2013, Dec - LSI acquired by Avago Technologies (press release)
• 2016, Feb - Avago acquired Broadcom Corporation and renamed itself to Broadcom Inc.
• 2017, Apr - shipped SAS3408, SAS3508 based Tri-Mode HBAs and RAID adapters (press release) / first controllers to support SAS/SATA/NVMe; moving from PowerPC to ARM A15

 

[nezach]

ICs

LSI makes several types of common architecture ICs that they sell to OEMs as well as use them to power their HBAs and RAID controller cards. Those ICs are known as: SAS/SATA Storage I/O Controllers (IOCs) and RAID-on-Chip ICs (RoCs).

• IOCs are PCI Express to SAS+SATA controllers with optional basic integrated RAID (IR) capability supporting RAID levels 0, 1, 1E, 10.
• RoCs are based on the same technology as IOCs, but integrate more advanced hardware based RAID functionality such as hardware acceleration engines for RAID 5 and RAID 6 parity calculations, memory controller, battery and/or supercap backup. They support RAID levels 0, 1, 1E, 5, 6, 10, 50 and 60.

SAS2308 IOC Block Diagram (source: SAS2308 product brief)


SAS2208 RoC Block Diagram (source: SAS2208 product brief)


Processor Cores

Every IOC and RoC includes several processor cores. These cores are used to implement Fusion-MPT architecture, Integrated RAID (IR) functions and completely manage SCSI I/Os without involving host CPU.

Early IOCs LSI53C1030, SAS1064, and SAS1068 used 32-bit ARM966E-S RISC processor. In the case of LSI53C1030 a total of 3 cores were used. One was used as an I/O processor (IOP) to control the system interface and manage the host side of non-DMA accesses to the Ultra320 SCSI bus. And two were used as context managers for controlling the SCSI side of data transfers, one for each channel. See chapter 2 of LSI53C1030 technical manual for more details.

LSI53C1030 Block Diagram (source: LSI53C1030 technical manual)


With the introduction of the first RoC SAS1078 LSI migrated from ARM 9 to PowerPC 440 processors, which they used for all SAS2 IoCs and ROCs. At that time AMCC owned PowerPC 440 assets as well as 3ware business. In 2009 LSI acquired 3ware from AMCC and later that year IBM announced that it has collaborated with LSI to develop successor to PowerPC 440, the PowerPC 476FP. LSI went on to use PowerPC 476 on SAS3 IoCs/RoCs SAS30xx, SAS31xx, SAS32xx, SAS33xx.

Starting with the tri-mode (SAS/SATA/NVMe) ICs: SAS3408 IOC and SAS3508 RoC LSI moved back to ARM architecture with ARM A15 processor.

PowerPC 400 - Wikipedia
The PowerPC 440 Core
IBM Announces Highest Performance Embedded Processor for System-on-Chip Designs
Introduction to the IBM PowerPC 476FP Embedded Processor Core
Embedded Symmetric MultiProcessing system on a SoC with 1.6GHz PowerPC IP in 45nm

 

[nezach]

Fusion-MPT

Fusion-MPT (Message Passing Technology) is an architecture, think umbrella term, which encompasses Fusion-MPT firmware, hardware (Ultra320 SCSI, FC, SAS) and OS level drivers. It was announced in the year 2000, about 2 years after Symbios acquisition and all IOCs and RoCs released since then are based on (or rather implement) this architecture.

Unfortunately technical details about this architecture are rather scarce and words Fusion-MPT can take on different meaning in different contexts, making it difficult to accurately explain what it actually is and/or does. Let’s take a closer look at each individual piece of this architecture.

Fusion-MPT Architecture Diagram (source: LSISAS_Invalid_Page_Analysis.pdf)


Hardware

At a hardware level Fusion-MPT is designed to support multiple physical interfaces as well as be easily extensible to new physical interfaces. First ever Fusion-MPT chip was FC909 fibre channel controller, followed by LSI53C1030 Ultra320 SCSI controller. Interestingly, PR documents of that era also talk about InfiniBand support, though I am not aware of any commercial InfiniBand product from LSI. SAS1064 was the first controller to add SAS support and SAS3408/SAS3508 introduced NVMe support.

Fusion-MPT MPI (Message Passing Interface)

Fusion-MPT MPI is probably the most important feature of the Fusion-MPT architecture. It provides an “abstraction” layer between host driver and Fusion-MPT chip. It is a standard communication protocol that defines how host driver and applications interfaces with all Fusion-MPT compatible devices. This enables single device driver to support all Fusion-MPT devices and makes drivers impervious to silicone or firmware changes. MPI can also be used for custom device driver implementation or value-added application programming, such as Integrated RAID.

MPI proper (Message Passing Interface - Wikipedia) is actually a standardized message-passing standard intended for use in parallel computing architectures e.g. supercomputers. It is maintained by a group of research and industry partners and was first released in 1994. Fusion-MPT MPI is implemented using MPI standard.

OS Driver and Firmware

OS level driver is responsible for sending MPI request messages to firmware and firmware replies with MPI reply messages after processing the request. This is a verbatim quote from one of the LSI’s technical docs that better explains this:

"The LSI SAS firmware is part of the Fusion-MPT device driver architecture. The role of the SAS firmware is to do an I/O request that is issued from a host driver, and to reply with a success or failure status. Note that the host driver could be a driver stack in an operating system (such as the Windows SCSI Port and Miniport drivers), or it could be RAID firmware running on an LSI controller or on an external IOP. A key part of the architecture is that the host driver is in control, and the firmware just does what the driver tells it to do. In this architecture, the host driver stack is responsible for most retry logic and timeouts for commands it sends. In most cases when an error is detected, the firmware replies with an error status to the host and allows the host driver stack to determine if it wants to fail the I/O to the OS, or retry it, or do something else. The Fusion-MPT architecture is a SCSI Command Set based architecture in which the host driver passes a message to the firmware that includes a SCSI command, and the firmware attempts to send the command to a target. Then the firmware replies to the host driver with a status of success or failure. Failure responses include SCSI Status and Sense Data from the target, if present."

MPI messages are used not only for SCSI I/O, but also for controller configuration and maintenance tasks. There are messages that can be used to reset the IOC, enable/disable port, flash firmware, configure RAID, etc. These maintenance messages are used not only by the driver, but also by the BIOS, WebBIOS, UEFI BSD as well as user space applications such as LSIUtil.

LSI Logic's Fusion-MPT Architecture (2007)
Fusion-MPT Device Management User Guide v1.3 (2007)
Fusion-MPT Device Management User Guide v1.2 (2002)
Fusion-MPT FCode User's Guide (2004)
LSISAS_Invalid_Page_Analysis.pdf <- This is an interesting document (in Chinese, use translator) that takes a deeper look into MPI messages, Linux kernel driver and few other things.

 

[nezach]

Technical Documents

Since LSI sells their ICs to OEMs, they must also provide OEMs with technical manuals for those chips. These days that documentation is not publicly available and most likely hidden behind NDAs. Fortunately things were simpler back in the day and LSI used to publicly post technical manuals for some of their chips. These documents are now more than a decade old, but the fact that new ICs are still based on the Fusion-MPT architecture means that a surprising number of details from those docs are still valid.

• LSI53C1030 PCI-X to Dual Channel Ultra320 SCSI Multifunction Controller Technical Manual v2.2 (2003)
  This manual for 53C1030 is a treasure trove of technical information. Some of the more interesting things that are covered in this doc are: subsystem level overview, Fusion-MPT architecture, functional description of interfaces, detailed explanation of MPI message flow, PCI address spaces, PCI host registers and their description.

Following guides are for LSI’s fibre channel Fusion-MPT controllers, but again, since they are Fusion-MPT based, a lot of the stuff applies to SCSI chips as well.

• LSIFC929XL Fibre Channel I/O Processor Technical Manual v2.0 (2005)
  Notable sections: Chapter 2 - nice intro into fibre channel; Section 3.2.2 - MPI message flow with a diagram (diagram is not present in the LSI53C1030 manual), Section 3.5 - memory interfaces
• LSIFC909 Fibre Channel I/O Processor A Technical Manual (2000)
  Largely older version of the LSIFC929XL manual, but has more detailed functional block diagram of the controller: figure 1.2

This document provides information on fault code decoding. It also explains how to decode heartbeat LED flashes when controller enters fault state.

• LSI SAS Error Codes (S11079) Systems Engineering Note v6.0 (2009)

Arguably the most important Fusion-MPT architecture document is the MPI specification guide. It covers the “software” side of the architecture. Surprisingly it is nowhere to be found online. There are a lot of different versions of the MPI spec document referenced in other documents. Some of them are:

• Fusion-MPT Message Passing Interface Specification, Volume 1.2, DB14-000174-02
• Fusion-MPT Host Driver Programming Guide, Volume 1.2, Document No. DB15-000203-01
• Fusion-MPT Message Passing Interface Specification, v1.5.6 (September 2007) DB14-000174-09
• Fusion-MPT 2.0 MPI Specification Guide
• Fusion-MPT 2.5 MPI Specification

Maybe someone with better Google skills will be able to find it.

 

[zxv]

Here is documentation on the current version of MPI.

Open MPI v4.0.1 documentation

I've written software that uses MPI for clustered parallel processing.

 

[nezach]

@zxv if I understand correctly Open MPI is just one of the MPI implementations, right? I don't have any experience with this stuff, but I briefly looked at the MPI standard and frankly I don't even know how exactly LSI's Fusion-MPT MPI relates to it. Probably need to take a look at the source code of mpt3sas Linux driver to get a better idea.

 

[zxv]

Yes, that's correct. Certain network card vendors, such as Intel and Mellanox, have their own MPI distributions based on openmpi that talks to their drivers. Those vendors will optimize their MPI implementation using RDMA or proprietary protocols specific to their hardware, to achieve lower latency.

You generally need only a couple of MPI calls, to set up the connection, and send/receive messages. Fusion-MPT might be using a very limited set of MPI API functions.

 

[nthu9280]

Great discussion, though quite a bit of it is way beyond my comprehension.

 

[nezach]

@zxv thanks for elaborating, it actually makes a lot more sense now. Do you know if parallelism comes into play when MPI is used for driver <-> firmware communication, after all we only have a single driver and a single firmware.

@nthu9280 I actually feel the same way , I am counting on people with deeper understanding than me to join and explain/clarify some of the things.

 

[zxv]

@zxv thanks for elaborating, it actually makes a lot more sense now. Do you know if parallelism comes into play when MPI is used for driver <-> firmware communication, after all we only have a single driver and a single firmware.

It seems a little strange to be using a networking protocol over PCI bus, but perhaps the driver is also designed to talk to devices over a SAS bus. That's pure speculation though.

Yes, MPI is designed explicitly for concurrent parallel message passing. It's highly efficient not just because all the various senders can act concurrently, but also because the message passing is explicit -- the application code has complete control over when messages are sent and can arrange to overlap communication and computation. That overlap is a key to maximizing cluster performance.

The API provides some synchronization functions, so that sender and receiver can wait till both reach a designated 'barrier'.

"open fabrics" are an evolution of high performance networking, of which MPI was an important part. "NVME over fabric" is an evolution toward networking speed and latency suitable for NVME storage. Vendors in this space also support MPI, so using MPI offers some future proofing.

 

[nezach]

Fusion-MPT MPI

Versions

There are two major versions of the Fusion-MPT MPI specification v1.x and v2.x; latter also has several minor revisions: v2.0, v2.5 and v2.6. Every IOC/RoC chip implements a specific major.minor version of the specification. Starting with MPI v2.5, features specific to that version can only be used on v2.5 products and must not be used on v2.0 products. Features specific to v2.0 can be used on both v2.0 and v2.5 products. Same applies to v2.6 products.

All Ultra320 SCSI and SAS 1 products are MPI v1.x, SAS 2 products are MPI v2.0, SAS 3 4/8 port chips are MPI v2.5, and SAS 3 16/24 port and all Tri-Mode chips are MPI v2.6.

IC to MPI version assignments are defined in the Linux mpt3sas driver, see _scsih_determine_hba_mpi_version in mpt3sas_scsih.c : .

• MPI v1.x - 53C1020 / 53C1030 / 53C1035 / SAS1064 / SAS1068 / SAS1078
• MPI v2.0 - SAS2004 / SAS2008 / SAS2108 / SAS2116 / SAS2208 / SAS2308
• MPI v2.5 - SAS3004 / SAS3008 / SAS3108
• MPI v2.6 - SAS3216 / SAS3224 / SAS3316 / SAS3324 / SAS3408 / SAS3508 / SAS3416 / SAS3516 / SAS3616W

Header Files

It is quite disappointing that we don’t have access to the Fusion-MPT MPI specification guide, but we do have access to the next best thing - C header files. What’s even better is that those files are released under GPL2 and BSD licenses. There is a separate set of header files for MPI v1.x and MPI v2.x. We are not really all that interested in MPI v1.x, but I am including those files here for reference. Best place to get MPI headers is open source OS kernels.

MPI v2.x
Source: Linux (latest), FreeBSD, Illumos. I marked files that are of most interest to us with *.

• mpi2.h* - MPI Message independent structures and definitions
• mpi2_cnfg.h* - MPI Configuration messages and pages
• mpi2_hbd.h - MPI Host Based Discovery messages and structures
• mpi2_image.h* - Contains definitions for firmware and other component images / starting with v2.6
• mpi2_init.h - MPI SCSI initiator mode messages and structures
• mpi2_ioc.h* - MPI IOC, Port, Event, FW Download, and FW Upload messages
• mpi2_pci.h - MPI PCIe Attached Devices structures and definitions.
• mpi2_ra.h - MPI RAID Accelerator messages and structures
• mpi2_raid.h - MPI Integrated RAID messages and structures
• mpi2_sas.h - MPI Serial Attached SCSI structures and definitions
• mpi2_targ.h - MPI Target mode messages and structures
• mpi2_tool.h* - MPI diagnostic tool structures and definitions
• mpi2_type.h - MPI basic type definitions
• mpi2_history.txt - Changelog

MPI v1.x
Source: Linux, FreeBSD (latest)

• mpi.h - MPI Message independent structures and definitions
• mpi_cnfg.h - MPI Config message, structures, and Pages
• mpi_fc.h - MPI Fibre Channel messages and structures
• mpi_init.h - MPI initiator mode messages and structures
• mpi_ioc.h - MPI IOC, Port, Event, FW Download, and FW Upload messages
• mpi_lan.h - MPI LAN messages and structures
• mpi_log_fc.h - MPI IocLogInfo definitions for the SYMFC9xx chips
• mpi_log_sas.h - SAS firmware interface IOC Log Info codes
• mpi_raid.h - MPI RAID message and structures
• mpi_sas.h - MPI Serial Attached SCSI structures and definitions
• mpi_targ.h - MPI Target mode messages and structures
• mpi_tool.h - MPI Toolbox structures and definitions
• mpi_type.h - MPI Basic type definitions
• mpi_history.txt - Changelog

 

[zxv]

Nice! I can really appreciate the effort to bring all the information and code references together. In the header files, are the C structs used to compose messages in memory, that are then sent/received using the MPI protocol?

 

[nezach]

Nice! I can really appreciate the effort to bring all the information and code references together. In the header files, are the C structs used to compose messages in memory, that are then sent/received using the MPI protocol?

The short answer is yes, my next post will go into detail about the transport mechanism.

 

[nezach]

Fusion-MPT MPI Message Interface

The Fusion-MPT architecture provides two I/O methods for the host system to communicate with the IOC: the system interface doorbell and the message queues. This post will cover the message queues that are used for normal operation and data transport.

There are two basic constructs in the message interface. The first construct, the message, is used to communicate between the system and the IOC. Messages are moved between the system(s) and the IOC using the second construct, a transport mechanism.

Messages

The IOC uses two types of messages to communicate with the system. Request messages are created by the system to "request" an action by the IOC. Reply messages are used by the IOC to send status information back to the system.

Request message data structures (aka system message frames) are up to 128 bytes in length. The message includes a message header and a payload. The header includes information to uniquely identify the message. The payload is specific to the request itself, and is unique for SCSI, and Target messages.

Message Flow



Before requests can be posted to the IOC, the system must allocate and initialize a pool of message frames (MFDs), and provide a mechanism to assign individual message frames, on a per-request basis. The host also must provide one message frame per target LUN, and prime the Reply Free FIFOs for each function with the physical address of these message frames. When allocation has been completed, requests flow from the host to the IOC, as follows:

1. The host driver receives an I/O request from the operating system.
2. The host driver allocates a system message frame (SMF) and builds an I/O request message within the SMF. The allocation method is the responsibility of the host driver.
3. The host driver creates the Message Frame Descriptor (MFD), and writes the MFD to the Request Post FIFO.
4. The I/O Controller (IOC) reads the MFD from the Request Post FIFO and DMAs the request to a local message frame.
5. The IOC sends the appropriate FC/SCSI/NVMe request, and subsequently receives the reply from the target.
   • If the I/O status is successful, the IOC writes the MessageContext value, plus turbo reply bits, to the Reply Post FIFO, which automatically generates a system interrupt. This is called Context Reply format.
   • If the I/O status is not successful, the IOC pops a reply message frame from the Reply Free FIFO, and generates a reply message in the reply message frame. The IOC then writes the system physical address of the reply message frame to the Reply Post FIFO, which generates a system interrupt. This is called Address Reply format.
6. The host driver receives a system interrupt and reads the Reply register. If there are no posted messages, the system reads the value of 0xFFFFFFFF.
7. The host driver responds to the operating system appropriately.
8. If the I/O status is not successful, the host driver returns it to the Reply Free FIFO.

Notes:

• Message frame descriptors (MFDs) are pointers to the system message frames (SMFs)
  
• MFDs are defined in mpi2.h header file
• Request message queue (request post FIFO) and response message queue (reply free FIFO and reply post FIFO) are located on SRAM. See LSI53C1030 Block Diagram in post #3.

Most of the information for this post was taken from section 3.2 of the LSIFC929XL technical manual and section 2.2 of the LSI53C1030 technical manual.

 

[nezach]

Fusion-MPT MPI Messages

Request Message Header

This is a definition of a generic request message header used for all request messages. Members FunctionDependent{1,2,3} (bytes 0x00, 0x01, 0x04, 0x05, 0x06) are reserved for request specific usage. E.g. firmware upload request uses byte 0x00 for ImageType, while configuration request uses it for Action.

typedef struct _MPI2_REQUEST_HEADER {
    U16 FunctionDependent1;   /*0x00 */
    U8 ChainOffset;           /*0x02 */
    U8 Function;              /*0x03 */  <-- Message function code; MPI2_FUNCTION_*
    U16 FunctionDependent2;   /*0x04 */
    U8 FunctionDependent3;    /*0x06 */
    U8 MsgFlags;              /*0x07 */
    U8 VP_ID;                 /*0x08 */
    U8 VF_ID;                 /*0x09 */
    U16 Reserved1;            /*0x0A */
} MPI2_REQUEST_HEADER, MPI2RequestHeader_t;

Message function code tells the IOC what action the request message is asking it to perform, or in other words what type of request is included in the payload.
^
This definition will be sufficient for our discussion, but strictly speaking it might not be completely accurate. One would expect every function code to map to a single request message (payload) type, but that is not the case. Function code MPI2_FUNCTION_TOOLBOX is used for multiple different payloads, one for each type of Tool code. I am including function code to request message struct mapping for reference.

Spoiler: Function Code to Request Struct Mapping

 - MPI2_FUNCTION_SCSI_IO_REQUEST               (0x00) MPI2_SCSI_IO_REQUEST | MPI25_SCSI_IO_REQUEST
 - MPI2_FUNCTION_SCSI_TASK_MGMT                (0x01) MPI2_SCSI_TASK_MANAGE_REQUEST
 - MPI2_FUNCTION_IOC_INIT                      (0x02) MPI2_IOC_INIT_REQUEST
 - MPI2_FUNCTION_IOC_FACTS                     (0x03) MPI2_IOC_FACTS_REQUEST
 - MPI2_FUNCTION_CONFIG                        (0x04) MPI2_CONFIG_REQUEST
 - MPI2_FUNCTION_PORT_FACTS                    (0x05) MPI2_PORT_FACTS_REQUEST
 - MPI2_FUNCTION_PORT_ENABLE                   (0x06) MPI2_PORT_ENABLE_REQUEST
 - MPI2_FUNCTION_EVENT_NOTIFICATION            (0x07) MPI2_EVENT_NOTIFICATION_REQUEST
 - MPI2_FUNCTION_EVENT_ACK                     (0x08) MPI2_EVENT_ACK_REQUEST
 - MPI2_FUNCTION_FW_DOWNLOAD                   (0x09) MPI2_FW_DOWNLOAD_REQUEST | MPI25_FW_DOWNLOAD_REQUEST
 - MPI2_FUNCTION_TARGET_ASSIST                 (0x0B) MPI2_TARGET_ASSIST_REQUEST
 - MPI2_FUNCTION_TARGET_STATUS_SEND            (0x0C) MPI2_TARGET_STATUS_SEND_REQUEST
 - MPI2_FUNCTION_TARGET_MODE_ABORT             (0x0D) MPI2_TARGET_MODE_ABORT
 - MPI2_FUNCTION_FW_UPLOAD                     (0x12) MPI2_FW_UPLOAD_REQUEST | MPI25_FW_UPLOAD_REQUEST
 - MPI2_FUNCTION_RAID_ACTION                   (0x15) MPI2_RAID_ACTION_REQUEST
 - MPI2_FUNCTION_RAID_SCSI_IO_PASSTHROUGH      (0x16) MPI2_SCSI_IO_REQUEST
 - MPI2_FUNCTION_SCSI_ENCLOSURE_PROCESSOR      (0x18) MPI2_SEP_REQUEST
 - MPI2_FUNCTION_SMP_PASSTHROUGH               (0x1A) MPI2_SMP_PASSTHROUGH_REQUEST
 - MPI2_FUNCTION_SAS_IO_UNIT_CONTROL           (0x1B) MPI2_SAS_IOUNIT_CONTROL_REQUEST
 - MPI2_FUNCTION_IO_UNIT_CONTROL               (0x1B) MPI26_IOUNIT_CONTROL_REQUEST
 - MPI2_FUNCTION_SATA_PASSTHROUGH              (0x1C) MPI2_SATA_PASSTHROUGH_REQUEST
 - MPI2_FUNCTION_DIAG_BUFFER_POST              (0x1D) MPI2_DIAG_BUFFER_POST_REQUEST
 - MPI2_FUNCTION_DIAG_RELEASE                  (0x1E) MPI2_DIAG_RELEASE_REQUEST
 - MPI2_FUNCTION_TARGET_CMD_BUF_BASE_POST      (0x24) MPI2_TARGET_CMD_BUF_POST_BASE_REQUEST
 - MPI2_FUNCTION_TARGET_CMD_BUF_LIST_POST      (0x25) MPI2_TARGET_CMD_BUF_POST_LIST_REQUEST
 - MPI2_FUNCTION_RAID_ACCELERATOR              (0x2C) MPI2_RAID_ACCELERATOR_REQUEST
 - MPI2_FUNCTION_HOST_BASED_DISCOVERY_ACTION   (0x2F) MPI2_HBD_ACTION_REQUEST
 - MPI2_FUNCTION_PWR_MGMT_CONTROL              (0x30) MPI2_PWR_MGMT_CONTROL_REQUEST
 - MPI2_FUNCTION_SEND_HOST_MESSAGE             (0x31) MPI2_SEND_HOST_MESSAGE_REQUEST
 - MPI2_FUNCTION_NVME_ENCAPSULATED             (0x33) MPI26_NVME_ENCAPSULATED_REQUEST
 - MPI2_FUNCTION_MIN_PRODUCT_SPECIFIC          (0xF0)
 - MPI2_FUNCTION_MAX_PRODUCT_SPECIFIC          (0xFF)
 - MPI2_FUNCTION_TOOLBOX                       (0x17)
   - MPI2_TOOLBOX_CLEAN_TOOL                   (0x00) MPI2_TOOLBOX_CLEAN_REQUEST
   - MPI2_TOOLBOX_MEMORY_MOVE_TOOL             (0x01) MPI2_TOOLBOX_MEM_MOVE_REQUEST
   - MPI2_TOOLBOX_DIAG_DATA_UPLOAD_TOOL        (0x02) MPI2_TOOLBOX_DIAG_DATA_UPLOAD_REQUEST
   - MPI2_TOOLBOX_ISTWI_READ_WRITE_TOOL        (0x03) MPI2_TOOLBOX_ISTWI_READ_WRITE_REQUEST
   - MPI2_TOOLBOX_BEACON_TOOL                  (0x05) MPI2_TOOLBOX_BEACON_REQUEST
   - MPI2_TOOLBOX_DIAGNOSTIC_CLI_TOOL          (0x06) MPI2_TOOLBOX_DIAGNOSTIC_CLI_REQUEST | MPI25_TOOLBOX_DIAGNOSTIC_CLI_REQUEST
   - MPI2_TOOLBOX_TEXT_DISPLAY_TOOL            (0x07) MPI2_TOOLBOX_TEXT_DISPLAY_REQUEST
   - MPI26_TOOLBOX_BACKEND_PCIE_LANE_MARGIN    (0x08) MPI26_TOOLBOX_LANE_MARGINING_REQUEST

Message Functions

Let's now take a look at what each message function does. It is worth noting that everything discussed up to this point applied to both IT/IR and MegaRAID (MR) firmware, but this is where things start to diverge. Most of these functions apply to IT/IR firmware only. Message function code is in included in brackets.

• MPI Configuration functions:
  • MPI2_FUNCTION_CONFIG (0x04) - Access or modify operational parameters (pages) of the IOC.
• MPI IOC, Port, Event, FW Download, and FW Upload functions:
  • MPI2_FUNCTION_IOC_INIT (0x02) - Move the IOC to the Operational state.
  • MPI2_FUNCTION_IOC_FACTS (0x03) - Obtain information (facts) about the IOC.
  • MPI2_FUNCTION_PORT_FACTS (0x05) - Retrieve port-specific information (facts) about a port on the IOC.
  • MPI2_FUNCTION_PORT_ENABLE (0x06) - Enable port on the IOC.
  • MPI2_FUNCTION_EVENT_NOTIFICATION (0x07) - Turn event notification on or off.
  • MPI2_FUNCTION_EVENT_ACK (0x08) - Acknowledge the receipt of an event.
  • MPI2_FUNCTION_SEND_HOST_MESSAGE (0x31) - Send a message to a host on another virtual function.
  • MPI2_FUNCTION_FW_DOWNLOAD (0x09) - Download a firmware, BIOS, NVDATA, or related image to the IOC's nonvolatile storage.
  • MPI2_FUNCTION_FW_UPLOAD (0x12) - Upload a firmware, BIOS, NVDATA, or related image from the IOC's nonvolatile storage.
  • MPI2_FUNCTION_PWR_MGMT_CONTROL (0x30) - Control power management features on the IOC.
  • MPI2_FUNCTION_IO_UNIT_CONTROL (0x1B) - Control the IO Unit. Perform reset on the PHYs of the IO Unit, enable device NCQ, removes device, etc.
• MPI diagnostic tool functions:
  • MPI2_FUNCTION_DIAG_BUFFER_POST (0x1D) - Post a diagnostic buffer to the IO Unit.
  • MPI2_FUNCTION_DIAG_RELEASE (0x1E) - Request the IO Unit to release a diagnostic buffer.
  • MPI2_FUNCTION_TOOLBOX (0x17) - Send a non-I/O tool request to the IOC.
    • MPI2_TOOLBOX_CLEAN_TOOL (0x00) - Erase non-volatile adapter storage: pages, BIOS, flash, etc.
    • MPI2_TOOLBOX_MEMORY_MOVE_TOOL (0x01) - Read local (chip) memory.
    • MPI2_TOOLBOX_DIAG_DATA_UPLOAD_TOOL (0x02) - Toolbox Diagnostic Data Upload request (?)
    • MPI2_TOOLBOX_ISTWI_READ_WRITE_TOOL (0x03) - Toolbox ISTWI Read Write Tool request message (?)
    • MPI2_TOOLBOX_BEACON_TOOL (0x05) - Flash LED on HBA
    • MPI2_TOOLBOX_DIAGNOSTIC_CLI_TOOL (0x06) - Access UART Debug Console
    • MPI2_TOOLBOX_TEXT_DISPLAY_TOOL (0x07) - Send a string to IOC's Console using different console types (eg: UART serial terminal or Ethernet terminal)
    • MPI26_TOOLBOX_BACKEND_PCIE_LANE_MARGIN (0x08) - Perform PCIe Lange Margining (measure performance of PCIe link?)
• MPI SCSI initiator mode functions:
  • MPI2_FUNCTION_SCSI_IO_REQUEST (0x00) - Initiate SCSI I/O on a target device. (Sends SCSI CDB to the firmware)
  • MPI2_FUNCTION_SCSI_TASK_MGMT (0x01) - Send a task management command to a target device. Target reset, LUN reset, etc.
  • MPI2_FUNCTION_SCSI_ENCLOSURE_PROCESSOR (0x18) - Communicate with an SES device. Controls LED status signals, etc.
• MPI Target mode functions:
  • MPI2_FUNCTION_TARGET_ASSIST (0x0B) - Continue I/O as a SCSI target.
  • MPI2_FUNCTION_TARGET_STATUS_SEND (0x0C) - Send status for an I/O as a SCSI target.
  • MPI2_FUNCTION_TARGET_MODE_ABORT (0x0D) - Abort a target mode request message.
  • MPI2_FUNCTION_TARGET_CMD_BUF_BASE_POST (0x24) - Post base address of command buffer pool to the IOC.
  • MPI2_FUNCTION_TARGET_CMD_BUF_LIST_POST (0x25) - Repost an IoIndex to the IOC after it has been used.
• MPI Host Based Discovery functions:
  • MPI2_FUNCTION_HOST_BASED_DISCOVERY_ACTION (0x2F) - Perform an action on a device instance during Host Based Discovery.
• MPI SAS functions:
  • MPI2_FUNCTION_SMP_PASSTHROUGH (0x1A) - Send SMP functions to a SAS expander.
  • MPI2_FUNCTION_SATA_PASSTHROUGH (0x1C) - Send SATA commands to an attached SATA device.
  • MPI2_FUNCTION_SAS_IO_UNIT_CONTROL (0x1B) - Control the SAS IO Unit. Perform reset on the PHYs of the IO Unit, enable device NCQ, removes device, etc. MPI v2.5 and earlier only. Replaced by MPI2_FUNCTION_IO_UNIT_CONTROL in MPI v2.6 and later.
• MPI PCIe Attached Devices functions:
  • MPI2_FUNCTION_NVME_ENCAPSULATED (0x33) - Send NVMe command to an attached NVMe device.
• MPI RAID Accelerator functions:
  • MPI2_FUNCTION_RAID_ACCELERATOR (0x2C) - Initiate a RAID Accelerator operation.
• MPI Integrated RAID functions:
  • MPI2_FUNCTION_RAID_ACTION (0x15) - Perform an action on or receive status information from an Integrated RAID volume. Create/delete volume, start/stop IR subsystem, etc.
  • MPI2_FUNCTION_RAID_SCSI_IO_PASSTHROUGH (0x16) - Send an I/O to the hidden physical disk of Integrated RAID volume.
• Reserved function code range:
  • MPI2_FUNCTION_MIN_PRODUCT_SPECIFIC (0xF0) - Beginning of the product-specific range of function codes.
  • MPI2_FUNCTION_MAX_PRODUCT_SPECIFIC (0xFF) - End of the product-specific range of function codes.

Majority of the message functions are used for maintenance, diagnostics, and management. The ones highlighted in red are the most relevant to us and we will explore them in more detail in later posts.

In the previous post we discussed how Fusion-MPT architecture supports multiple SCSI transport protocols and physical layers such as fibre channel, parallel SCSI, iSCSI, SAS, but hardware and firmware abstracts this away from the driver. We can now see that from the host driver perspective it only needs to use a single MPI message function MPI2_FUNCTION_SCSI_IO_REQUEST to issue SCSI I/O commands without actually being aware of the flavor of the device.

 

[nezach]

HBAs and RAID Controller Cards

LSI makes two different types of storage cards: host bus adapters (HBAs) and RAID controller cards.

• HBAs always use IOC chip (well there is that oddball 9400-8i8e that uses RoC), but come with two types of firmware known as IT and IR.
• RAID controller cards (MegaRAID) use either IOC or RoC chips.
  • MegaRAID cards targeting entry level market use IOC chips and hardware wise are almost identical to the HBAs using same IOC, except that they come with MegaRAID firmware. These cards do not have onboard cache memory, battery backup and do not support RAID levels 6 and 60, they also have limitations on stripe unit size and other things. These cacheless MegaRAID cards are also known as iMR or iMegaRAID (Integrated MegaRAID) cards.
  • MegaRAID cards targeting value and higher market use RoC chips, come with onboard cache memory and support advanced features such as battery backup, RAID levels 6, 60, etc.

See LSI storage authority software user guide for MR, iMR, IR, and IT software feature comparison matrix.

A lot of OEMs sell HBAs and RAID controller cards powered by LSI’s ICs under their own brand. These cards usually come in several flavors:

• Some are simply re-branded LSI cards. They are manufactured by LSI and are physically identical (might have small differences) to LSI cards. It is easy to identify them because they usually sport LSI/Avago/Broadcom logo on the PCB e.g. IBM ServeRAID M1015 is a re-branded MegaRAID SAS 9240-8i.
• Other cards use LSI’s ICs, but are custom designed and manufactured. Most Dell PERC, SuperMicro and Fujitsu cards fall into this category.

 

[james23]

oh wow is this an excellent thread! exactly the types of info and detail ive been looking for, for quite some time! (up to about a yr or so ago i mainly used adaptec HW raid cards so navigating the various LSI models and capabilites/speeds took some learning, but worth it as they are great cards)

thanks so much nezach for taking the time to do this!

i wanted to cross ref these good , related links as well (a list of specific LSI card models):
https://forums.servethehome.com/ind...and-hba-complete-listing-plus-oem-models.599/

and this one (where nezach has lead the way), on trying to get low "ebay" cost lenovo 530-8i (9440-8i in hardware) cards to flash to LSI Raid FW (no success so far) or also flashing them to LSI IT FW ( IT flashing has been successful, to get you a low $ sas3 HBA):
https://forums.servethehome.com/ind...3408-got-myself-a-530-8i-on-ebay.21588/page-6

sorry i cant add more, but if i have some relevant data/info i will try to add to this post. thanks again

 

[bruce.b]

Hi All,

Wow, a lot of good stuff. I have a simple problem that maybe someone knows. I have many SAS3080X cards based on the 1068E chip. It has 8 ports. I an setting the card up with LSIutil. I connect the drives to cable connectors 1, 2. In some menus it shows up as Phy0 and Phy1 which is great. However the ID (which becomes SCSI ID) changes between boards. Some are 1,2; others are 8,9 or 10,11. I want the all to be 0,1. Looks like I have to go through the MPT to change this. Any help would be appreciated.

 

[Net-Runner]

Perfect thread. Thanks for all the information! It is really great to be able to dive deep into the background of things we are used to work with.