INTRODUCTION
PCI Express® (PCIe®) is a dominant technology for hardware applications requiring high-speed connectivity between networking, storage, FPGA, and GPGPU boards to servers and desktop systems. It is a robust technology that has evolved over decades to keep up with advancements in throughput and speed for I/O connectivity for computing requirements.
For memory-intensive and high-performance computing, Direct Memory Access (DMA) is an indispensable application. The trend over the years has been to move the DMA controller into devices using a point-to-point bus architecture to reduce latency and increase memory access throughput. A typical DMA operation in PCIe is the transfer of data from the system memory - that the host has access to - to end point devices.
This article discusses how verification engineers can use Mentor’s Questa® Verification IP (QVIP) to improve productivity during the functional verification of PCIe designs with DMA engines.
PCIe is built upon a layered architecture consisting of a transaction layer for payload transfers, a data link layer for link management, and a physical layer for initialization and training of a reliable PCIe link between two devices. In terms of PCIe verification, each layer has its own challenges and complexities. For verification of DMA engines, the verification effort is concentrated on the data transfer aspect of PCIe, which resides in the transaction layer. This article mentions techniques to speed up the PCIe link training and initialization processes as well as PCIe device enumeration in order to reduce the initial simulation runtime required to set up tests targeted towards verification of DMA functionality.
The following sections address the processes used to speed up initial PCIe set up and reference a use case where QVIP is used with the PCIe controller for DMA applications from PLDA, a developer of semiconductor intellectual property specializing in high-speed interconnects.
- Initial PCIe link training and speed negotiation process: PCIe link up is a prerequisite phase in all tests exercising data transfers across PCIe link. Reduction in simulation runtime in this phase boosts verification productivity by having a shorter runtime for tests focused on verification of design features.
- PCIe device enumeration and configuration space set up process: PCIe device discovery process (aka, device enumeration) is performed by the root port to discover end point device capabilities and set up the device and DMA engine after link up. A shortened set up time after link up also assists with reduced simulation runtime.
- QVIP use case with the PLDA PCIe controller for DMA applications: PLDA chose Mentor QVIP to test standard compliance for PCIe and AMBA/AXI in their XpressRICH-AXI product line. In Mentor QVIP, PLDA found a flexible and reliable tool for building its proprietary test suite on a highly-scalable testbench.
INITIAL PCIE LINK TRAINING AND OPERATING SPEED NEGOTIATION
The essential step in a functional test using PCIe is to perform PCIe link training and initialization before data transfer can commence between the two PCIe devices. This step is an integral part of every test that utilizes PCIe for data transfer. Optimizing PCIe link up will result in a reduction in the simulation runtime to reach the L0 state (the fully-operational link state for data transfer). Figure 1 on the following page shows the states in the Link Training Status State Machine (LTSSM) that gets executed by the two devices when negotiating a PCIe link. The four main states through which the LTSSM traverses in order to establish a reliable PCIe link are Detect, Polling, Configuration, and Recovery (Figure 1). The PCIe link traverses these four main states (and various sub-states defined within them) starting from Detect and following the path shown by the highlighted arrows to reach the L0 state.
 |
Figure 1 - PCIe link training state machine
During traversal of the LTSSM, the two PCIe devices exchange training sequences to negotiate a number of link parameters; including elements such as lane polarity, link/lane numbers, equalization, data rate of operation, and so on. Each of the main states define a set of counters for transmitting and receiving these training sequences in order to calibrate the transmitter and receiver of the two devices while advertising their link capabilities to form a reliable link based on mutual negotiation. These LTSSM transitions are mandated with a fixed number of training sequences ordered sets that need to be transmitted and received in these states.
Alongside these counters, there are timeouts defined per state and sub-state to avoid LTSSM deadlock issues that may occur due to transmitter/receiver errors and reset the LTSSM to the detect state.
The default values of these counters and timeouts are set for establishing a PCIe link in real hardware. In a simulation environment for RTL verification, these counters and timeouts need to be scaled down such that a reliable PCIe can be formed within an optimal simulation time with the desired link operation speed.
PCIE GEN5 Introduces An Optional Link Equalization Bypass Mode
Aside from reducing the counter values and scaling down the timeout, PCIe GEN5 introduces an optional link equalization bypass mode for faster link up. To train PCIe link at 32 GT/s, a conventional speed change process comprises: initially training the link to L0 at 2.5 GT/s and then initiating speed change followed by link equalization at the intermediate speeds: 8 GT/s, 16 GT/s, and finally 32 GT/s. Since equalization at data rates equal to or greater than 8 GT/s is an essential process for higher link reliability and lower bit error rates, the time spent performing speed change and equalization at each speed consumes approximately 100ms of simulation runtime.
With the equalization bypass mode, the PCIe link in L0 at 2.5 GT/s directly transitions the link speed to 32 GT/s. This process eliminates stepping through the intermediate data rates of 8 GT/s and 16 GT/s.
There are two variants of the optional equalization flow introduced in PCIe GEN5.
Equalization Bypass to 32 GT/s
- PCIe GEN5 introduces the ability to bypass equalization at the highest data rate, thereby removing the requirement of performing equalization at every data rate ≥ 8 GT/s. Negotiating this option during LTSSM means that there is no equalization needed in 8 GT/s or 16 GT/s. If the link works in 32 GT/s and then has reliability issues, then the link will need to perform equalization at 8 GT/s and then 16 GT/s.
- Within the equalization process there are effectively three main sub-phases - Phase1, Phase2, and Phase3 -for a downstream port and an additional phase - Phase0 - for an upstream port. If the downstream port is satisfied by the Phase1 equalization outcome, then it can skip Phase2 and Phase3 and complete the equalization process. This further reduces the simulation time spent in equalization while ensuring link reliability.
No Equalization Needed
- The option to not perform equalization at all is another simulation time saver. In this case the equalization parameters for transmitters and receivers are applied from a previously negotiated equalization process. These parameters are stored in persistent storage when equalization is once performed. These values are then applied directly during link reset such that equalization is not needed at all and link reliability is not compromised because no equalization was performed during LTSSM.
When verification engineers develop a testbench environment specifically for verifying DMA features, it is crucial that they configure the LTSSM parameters of the DUT and the configuration settings of the verification IP used in the testbench in-sync so that both devices can successfully transition the LTSSM states in step with each other and achieve PCIe link up in a reduced amount of simulation time.
Fine tuning these configuration parameters for both devices can become quite cumbersome and is an error-prone task, especially if these parameters are not known to the testbench developer. In this case, having a DUT and verification IP that provide a highly configurable design component becomes an absolute necessity to achieve the desired optimization.
PCIe QVIP provides a well-documented standard set of APIs to access LTSSM related configuration variables which include training sequences OS counters, timeouts, and the ability to configure LTSSM state and sub-state specific timeout configurations. With the ability to configure a varied set of LTSSM parameters in QVIP, it is imperative to keep the use-model as simplistic as possible. For ease of use, the default settings of these configurations are chosen such that the QVIP achieves an optimized LTSSM transition for PCIe link up. Having a highly configurable QVIP and optimized default setup greatly improves the usability of the Verification IP in a testbench.
PCIE DEVICE DISCOVERY AND CONFIGURATION SPACE SET UP PROCESS
This section elaborates on the PCIe device discovery process and configuration space set up performed by the host software from the root port in order to configure the end point device and DMA engine after the PCIe link is established. This step of device configuration is called the enumeration process. This step requires a series of Type0 configuration read and write TLPs issued by the root port in order to assign a bus number and device id to uniquely identify the end point device and configure the configuration space registers for all physical functions present in it.
In a functional verification environment, QVIP is configured as a root port connected to the PCIe DUT. The enumeration process is a lengthy sequence of configuration reads and writes that the QVIP performs. With QVIP built-in features and capabilities, this process can be reduced significantly by reducing the number of configuration reads and writes after link up, resulting in shorter simulation runtime set up.
Let us now inspect the enumeration process in more detail. For a PCIe topology where a root port is connected to an end point device, the enumeration process begins with sending a configuration read request - with bus number = 1, device number = 0, and function number = 0 - to read the vendor ID of the end point device. On receiving a successful completion for this, the configuration read confirms that there is a device connected to the root port.
The next step is to discover the type of device connected, by reading the header type field in the configuration space registers (shown in Figure 2 below). For an end point device the header layout field of the header type register should read Type 0 Configuration Space Header. This confirms that the device connected to the root port is an end point device. The multi-function device field of this register indicates that the device may contain multiple physical functions.
 |
Figure 2 - Configuration space registers per physical function in a PCIe device
From this point on, the enumeration process accesses the 4KB configuration space registers for each physical function of an end point device to determine the PCIe capabilities supported by a physical function as well as the memory resource requirements of the physical function by the host.
The steps below describe QVIP’s enumeration flow for end point device discovery.
- Read all the base address registers (BAR), starting at offset 10h, to determine the memory space region and the memory size requirements for this physical function.
- Read the capability pointer at offset 34h. PCIe uses a link-list structure to access the standard device capabilities and extended capabilities registers supported by a physical function. Here is a list of essential capabilities defined in the configuration space registers of a physical function (note: this is not a complete list of capabilities structures):
PCIe Base Capabilities
- PCI Power Management Capability Structure
- PCI Express Capability Structure
- MSI/MSI-X Capability Structures
Extended Capabilities
- SR-IOV
- Advanced Error Reporting
- Power Budgeting
- Latency Tolerance Reporting
- L1 PM Sub-states
- Enhanced Allocation
- Physical_Layer_16GT/s
- Lane Margining Receiver
- Physical_Layer_32GT/s
- Alternate Protocol
- After reading the complete capabilities list present in the configuration space registers by traversing the nodes of the linked list, the enumeration sequence follows a similar approach for physical functions supporting SR-IOV. The SR-IOV capability defines a set of lightweight PCIe functions, called virtual functions, that share one or more physical resources with the physical function. The enumeration sequence then follows a similar process of discovering the virtual functions supported by the physical function.
- Finally, the enumeration sequence now starts configuring the device by issuing a series of configuration writes to set up the device, based on the settings provided by the user in the QVIP agent configuration and the capabilities it discovered. This series of configuration writes are targeted specifically to set up the following:
- Initialization of BAR addresses for all the physical functions and virtual functions based on its memory requirements.
- Initialization of different device capabilities like power-management, max-payload size, maximum read request size, and read completion boundary.
- Enable bus-mastering capabilities of the device to initiate transactions on the PCIe bus.
- Initialization of MSI/MSI-X addresses for the devices.
While executing the enumeration sequence, the QVIP is also maintaining a data structure per the physical function it contains in order to utilize this information for user-specific test scenario creation after the enumeration is complete. This feature enables QVIP to easily execute extensive verification scenarios based on design capabilities, by providing the test writer with APIs to query design capabilities and provide address offsets for updating the configuration space registers in the device.
The number of configuration transactions executed in the enumeration sequence has a multiplication factor dependent on the number of physical functions and virtual functions per device. This setup phase needs to be performed for every test that uses the PCIe link for verifying DMA functionality. As a result, simulation runtime increases before any actual user-specific test scenario is executed. Reducing the simulation runtime by lowering the number of configuration transactions, significantly improves the set up time needed for a device.
QVIP provides two verification capabilities for enumeration sequences to reduce runtime dramatically.
Fast Enumeration
In fast enumeration mode, the QVIP is configured through a backdoor mechanism while the DUT is configured through configuration writes only. The advantage here is that the configuration reads for the configuration space registers do not take place, instead the QVIP does configuration writes to configure and set up the device. In this mode, runtime is reduced by half or even more (since configuration writes are fewer in number than the configuration reads performed during the enumeration sequence).
In this mode, configuration reads are not performed by QVIP. Still, the device capabilities information and memory resource requirements to perform configuration writes are needed. This crucial information is provided to QVIP using built-in utilities to accurately capture the required settings in a testbench usable format. The following are the steps needed to extract this information and feed it back into QVIP.
1. Run a test case, with default full enumeration mode setting:
<agent cfg handle>.agent_descriptor.auto_bring_up .enum_mode = PCIE_FULL_ENUM;
2. Enable the following commands to capture the configuration space register settings of the device:
<agent cfg handle>.agent_descriptor.auto_bring_up .bus_enum_setting .print_fast_bus_enum_setting = 1'b1;
3. Run the test and then open the simulation log to find the setting captured by QVIP, which will be used in the configuration phase of the test:
#// ||FAST_BUS_ENUM CONFIGURATION|| => Add the below configurations in the test file to enable the fast bus enumeration
#// ||FAST_BUS_ENUM CONFIGURATION|| => To enable fast bus enumeration mode enable the cfg.agent_descriptor.auto_bring_up.enum_mode = PCIE_FAST_ENUM
#// Configuring the extended cap offset for BDF = 00000100
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_AER_EXTD_CAP] = 'h100;
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_DEVICE_SERIAL_NUM_EXTD_CAP] = 'h1bc;
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_PWR_BUDGET_EXTD_CAP] = 'h280;
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_MF_VC_EXTD_CAP] = 'hf00;
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_VC_09_EXTD_CAP] = 'h148;
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_VENDOR_SPEC_EXTD_CAP] = 'h324;
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_ACS_EXTD_CAP] = 'h290;
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_ATS_EXTD_CAP] = 'h850;
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_DPA_EXTD_CAP] = 'h3fc;
...
#// ||FAST_BUS_ENUM CONFIGURATION|| =>ENDS==============================
The output between the banner’s FAST_BUS_ENUM CONFIGURATION in the simulation log is directly copied into the testbench for configuring the QVIP through the backdoor. Once the above settings are applied, the test case configuration is complete and ready to run in fast enumeration mode.
Providing QVIP with the above settings ensures that no further configuration read is necessary for accessing device capabilities. QVIP will now only perform the necessary configuration writes needed to set up the device for normal operational mode.
Backdoor Enumeration
In backdoor enumeration mode, configuration reads and writes are not performed at all. Configuration space registers for QVIP and the device are configured through a backdoor mechanism. The enumeration sequence in this mode is not performed.
This feature is dependent on the DUT to be able to update the configuration space register settings through a backdoor mechanism before the link is trained. PCIe design IP built with this capability can take advantage of this feature in QVIP and reduce the initial simulation runtime even further, as compared to fast enumeration.
The steps to extract the configuration space settings are similar to the fast enumeration mode with minor updates in the configuration option assigned to the PCIe QVIP agent.
1. Run a test case with the default full enumeration mode setting:
<agent cfg handle>.agent_descriptor.auto_bring_up .enum_mode = PCIE_FULL_ENUM;
2. Enable the following commands to capture the configuration space register settings of the device:
<agent cfg handle>.agent_descriptor.auto_bring_up .bus_enum_setting .print_bk_door_enum_setting = 1'b1;
3. Run the test and then open the simulation log to find the setting captured by the QVIP, which will then be used in the QVIP configuration phase of the test:
#// ||BACKDOOR CONFIGURATION|| => Add the below configurations in the test file to enable the backdoor enumeration
#// ||BACKDOOR CONFIGURATION|| => To enable backdoor enumeration enable the cfg.agent_descriptor.auto_bring_up.enum_mode = PCIE_BKDOOR_ENUM
#// Configuring the extended cap offset for BDF = 00000100
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_AER_EXTD_CAP] = 'h100;
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_ACS_EXTD_CAP] = 'h290;
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_ATS_EXTD_CAP] = 'h850;
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_SRIOV_EXTD_CAP] = 'ha04;
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_MULTICAST_EXTD_CAP] = 'h388;
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_PAGE_REQ_EXTD_CAP] = 'h770;
#rc_agent_cfg.m_device_cfg_space['h0100].pcie_extended_capability_offset[PCIE_DPA_EXTD_CAP] = 'h3fc;
...
#// ||BACKDOOR CONFIGURATION|| => ENDS=========================
For this mode, output between the banner’s BACKDOOR CONFIGURATION in the simulation log is directly copied into the testbench for configuring the QVIP through the backdoor. Once the above settings are applied, the test case configuration is complete and ready to run in backdoor enumeration mode.
Providing QVIP the above settings ensures that no configuration read is necessary for accessing the device capabilities, and QVIP assumes that since the user is running the test with the backdoor enumeration option, the configuration write is also not necessary.
When running QVIP in this mode, after the link is established the user can start initiating test scenarios with the assurance that QVIP and the device have completed the enumeration process.
To summarize, initial PCIe link training and the enumeration process is an essential part of every test for verification of DMA engines using PCIe. With the techniques described in the preceding sections, the simulation runtime required to establish an operational PCIe link at 32 GT/s is drastically reduced. Furthermore, the new equalization bypass mode in the PCIe GEN5 specification also helps reduce link training time. These optimizations ensure that the PCIe design set up is not compromised and does not impact the verification capability of the testbench. On average, taking advantage of the most optimized settings in a QVIP assisted testbench, the simulation runtime to establish a PCIe link is reduced by twenty percent. In one such typical case, link training time was reduced from 61 microseconds to 13 microseconds. This reduction in simulation runtime boosts the productivity of the verification engineer developing tests and reduces the overall turnaround time for debug and analysis.
QVIP USE CASE WITH THE PLDA PCIE CONTROLLER FOR DMA APPLICATIONS
For 20 years, PLDA has been a pioneer of PCIe technologies. PLDA created the XpressRICH-AXI product line a few years back, and they now propose a fifth generation, running at PCIe 5.0 speed (32 GT/s).
XpressRICH-AXI is a configurable and scalable PCIe controller Soft IP, as illustrated in figure 3 below. It is designed for ASIC and FPGA implementation, which is compliant with the PCI Express 5.0, 4.0, and 3.1/3.0 specifications. The IP can be configured to support endpoint, root port, and dual-mode topologies, allowing for a variety of use models, and it exposes a configurable, flexible AMBA AXI interconnect interface to the user. Users may optionally enable the built-in DMA engine, or connect an external DMA engine, such as PLDA's vDMA-AXI DMA, depending on their application requirements.
 |
Figure 3 - XpressRICH-AXI diagram
The extreme configurability and scalability of PLDA’s XpressRICH-AXI sub-system IP raised their verification challenges. PLDA decided to use Mentor QVIP to test standard compliance for PCIe and AMBA/AXI. With Mentor QVIP, PLDA discovered a flexible and reliable tool for building its proprietary test suite on a highly-scalable testbench.
 |
Figure 4 - Mentor QVIP instantiation in the PLDA UVM environment
 |
Figure 5 - XpressRICH-AXI detailed diagram
For the XpressRICH-AXI verification, PLDA uses the following Mentor QVIP features:
- Up to 64bit PIPE width (different widths at different link speeds)
- Up to 16 PCIe lanes
- Up to PCIe 5.0 speed
- PLDA MAC is ARI device => PLDA supports 32 physical function number
- SR-IOV
- Power management
- External clock and reset
For DMA applications, XpressRICH-AXI implements up to eight highly-configurable scatter-gather DMA engines. These engines can transfer data between PCIe and AMBA AXI4 interfaces. This functionality is itself a challenge, as PCIe transaction layer features need to be managed, with certain information carried in AXI4 packets. The DMA engine must take into account the PCIe Max Payload and Max Read Request sizes, as well as the BME bit in end point device mode. The DMA must also handle the number of outstanding requests via tag management. Some TLP parameters may need conversion.
For DMA verification, PLDA uses the following QVIP features:
- SR-IOV – DMA engine resources are sharable between multiple physical functions/virtual functions
- MSI/MSI-X – at the end of DMA or at error occurrence (per physical function/virtual function)
- Checking MSI data and Requester ID
- PASID prefix support
- Carrying PASID information
- ECRC generation/check
- Error Injection scenarios
- Errors occurred while fetching SG descriptor or data: completion with UR/CA status; completion timeout; poisoned completion; completion with ECRC error
- Reporting of error events in AER
- Checking AER registers
- Check the error messages sent to host by an end point device
- Testing the affect of FLR on DMA traffic (ongoing)
- Fast bus enumeration sequence for multiple implemented functions
- Scaled FC feature is mandatory for devices supporting GEN4 or above:
- Using scaled credits to size RX buffer
- Checking UFC DLLPs with QVIP
- 10-bit tag feature is mandatory for devices supporting GEN4 or above:
- Checking the TLP tags with QVIP
Example scenario of error detection and logging:
- DMA fetches a descriptor from PCIe domain by issuing an MRD TLP
- QVIP returns a completion which is poisoned (end point bit set)
- PCIe controller detects the error and: reports to DMA, logs the error in AER, sends error message if end point
- DMA sends MSI/MSI-X to report that error has occurred (using specific MSI vector)
- Continued operation of DMA is either permitted or not (depending on DMA transfer parameters)
CONCLUSION
XpressRICH-AXI combines a PCI Express 5.0 controller with complete AXI interconnect and DMAs. This complex architecture requires a highly-scalable and configurable testbench for verification. The high flexibility of Questa VIP was key to creating custom testbenches from scratch that can dynamically adapt to the different IP topologies and configurations, mixing PCIe interfaces with multiple AXI interfaces. PLDA and Mentor have collaborated throughout the different PCIe generations, enabling advanced leadership on the latest technologies together.
Back to Top