Verification planning requires identification of the key features from the design specification along with prioritization and testing of the functionality that leads to the development of a coverage model.
Functional coverage is an integral part in verifying the completeness of an IP. The key to functional coverage is to make an exhaustive verification plan containing coverpoints and crosses which are extracted from the protocol specification. This article explains the need for coverage-driven verification for Non-Volatile Memory Express (NVMe), and how QVIP achieves it.
WHY IS FUNCTIONAL COVERAGE IMPORTANT FOR NVMe
NVMe is an interface specification for PCI Express® based solid state drives focusing on low latency, scalability, and security. NVMe is a host controller interface and storage protocol that utilizes high-speed data transfer between enterprise and client systems and SSDs over a computer’s high-speed PCIe bus; i.e., it is an application layer.
Figure 1 - NVMe Over PCIe:
Key NVMe attributes:
- Support for multiple namespaces. Namespaces are quantities of non-volatile memory that may be formatted into logical blocks and referenced using unique Namespace IDs (NSID)
- Queue mapping. For admin queues there is always 1:1 mapping between the Submission Queue and Completion Queue and for I/O queues there can be 1:1 as well as n:1 mapping; i.e., N submission queues can be associated with a particular completion queue
- Support for 64K I/O queues
As design complexity increases debugging a bug in the design code can become quite difficult. This can be avoided by making a detailed test plan covering all possible scenarios. Achieving 100% functional coverage would ensure that the testing has been completed in accordance with the test plan. An NVMe subsytem can be comprised of multiple controllers; thereby increasing the complexity of verification as design behavior is controller specific. Therefore, coverage metrics play an important role in NVMe as coverage should come out to be the same for all the controllers.
A coverage model is created for the bus traffic; i.e., NVMe transactions driven on a PCIe bus. Therefore coverage helps in finding bugs in the monitor component and helps in monitoring/testing of assertions as well.
WHAT MUST BE COVERED IN NVMe (INCLUDED IN QVIP TEST PLAN)
This next section gives a detailed explanation for items which need to be covered as part of functional coverage of NVMe and already included in the QVIP test plan and code.
Controller Registers Coverage
NVMe defines a register map for the controller indicating controller capabilities, doorbell registers for queues, controller configurations, controller memory buffer location/size registers, and more. Without the proper values configured to each of the controller registers, the design can crash. Therefore, a separate covergroup should be made covering all possible values for each of the registers and to check for design behavior.
Common scenarios for covering the controller registers can be as follows:
- Default values configured to each of the registers as defined by the NVMe specification
- Verify that each register returns to its default values after reset
- Different types of arbitration mechanisms supported for command set
- Verifying whether reserved values of registers return zero only when read
- Ensure that application software (host) doesn’t access any region of memory of two or more registers
- Transition bins should be made for value of registers (e.g., CC.EN)
The above coverpoint is part of the QVIP controller registers covergroup and covers all possible values of the enable bit as part of controller configuration (CC) register values. This scenario helps us verify the controller in ways which are:
- Processing of commands and posting of completions
- Proper design functionality in case of a controller level reset
Commands and Its Responses Coverage
NVMe includes an admin command set and a NVM command set (I/O command set). Command covergroup is responsible for sampling all possible commands that can be processed through the NVMe controller, resulting in testing of the controller functionality. Stimulus is generated from the host side (BFM), which is processed by the controller. These results give information to the user about the tests which drive the DUT (controller).
Coverpoint: Admin opcodes
Bins are comprised of all possible admin opcodes—like Abort command, Get log page command, Set/Get feature, and Identify commands, etc.—which when sampled at the appropriate sampling condition would ensure correct behavior while processing of respective commands. The most important aspect here is the responses posted by the controller, which are covered with the help of cross coverage.
Coverpoint: Generic specific status values
Bins comprise all possible generic status values thereby ensuring that the controller is capable of returning all possible generic status values. Same goes for Command specific and Media Error status values.
Cross: Opcodes (Admin/IO) with status values. This would result in all status values being exercised, which can be posted by the controller for the respective command.
Data Transfer Mechanisms
NVMe supports two types of data structures for data transfer with commands:
- PRPs (Physical Region Page) and
- SGLs (Scatter Descriptor Lists)
Here, functional coverage would mean to check validity of these mechanisms with all valid commands. For the PRP case all the values in PRP1 and PRP2 entries should be exercised, and for SGLs all type of descriptors should have been used for all I/O commands involving data transfer.
Data transfer coverage will help users to know how data transfer takes place from the host to controller or vice versa, how PRPs or SGLs work, and check whether the DUT is functioning according to specification.
Some erroneous scenarios which should be observed while sampling could be:
- SGLs used with admin commands
- PRP2 list used with admin commands
QVIP takes care of all these combinations such as:
Covers whether prp1 values have been exercised with all valid admin opcodes.
Covers whether all types of SGL descriptors have been used with valid I/O commands.
Submissions Queue/Completion Queue Entry Fields
Submission queue entry fields should be covered for all types of commands, such as NSID used for valid commands and BROADCAST value of NSID used with valid commands. The most important thing to cover are all possible command specific status values for all possible commands (Admin as well as I/O).
Coverpoint: Log page identifiers (LID) covering all possible LID values
Coverpoints: Feature identifier (FID) values and their cross with Set/Get feature opcodes ensuring each possible FID for both the commands.
Completion posted for each type of command should be covered; i.e., all the entries in CQE should be checked for proper design functionality. Cross coverage here would be the most important aspect so as to exercise these fields for all types of possible commands. Scenarios to be covered can include:
- Do not retry and more bit values for all possible commands, and if more bit is set to 1, get log page with error information log should be retrieved for checking of additional information associated with the command
- All status code values of all possible status code types; i.e., generic, command specific, and media specific error values should be covered for all valid commands. Here one has to be careful in checking for which status values are valid for each kind of command
- Erroneous scenarios for reserved status code received
- Transition bins for values of phase tag value while posting CQE
Queue Related Scenarios
Queues are the most important aspect for command processing in NVMe. Test plan should take in account the following scenarios for proper design functionality:
- Wrapping condition for queues (circular in nature)
- Maximum and minimum depth of queues
- Each submission queue can be mapped to one or more completion queue. Both of these mappings should be covered as part of design functionality
- Erroneous scenarios such as deletion of I/O submission queues before corresponding completion queue
QVIP incorporates coverpoints for each of the above scenarios and offers various queue related configurations in the testbench for design testing purposes.
The NVMe controller includes various optional features such as virtualization management, Sanitize Actions as part of Sanitize Command, optional asynchronous event types such as firmware activation notices, etc. These optional features if supported by the controller should be covered as part of the design. One thing of significance is that if these features are not supported by the controller then all the coverpoints/crosses for that optional feature including command specific fields, actions, feature identifiers, and log page ids should be disabled; i.e., weight and goal should be zero.
A covergroup for these optional features can help the user find the missing features in the DUT.
QVIP allows for user defined configurations for optional coverpoints/crosses so as to ignore coverage if a design doesn’t support a particular feature.
NOTE: QVIP covers all the above scenarios by means of various covergroups which are created on a per controller basis in nvme_coverage class.
HOW QVIP IMPLEMENTS FUNCTIONAL COVERAGE
Test Plan Format
The most important aspect to verify any design is to make a functional verification plan containing the items to be verified in design. The format shown in Figure 2 is applied by QVIP when making a test plan.
Figure 2 - QVIP Test Plan Format:
Coverpoints/crosses are to be extracted from the protocol, and to ensure correct design functionality corner cases should also be taken in consideration. The main things to keep in mind while making a verification plan are:
- What to cover?
- When to sample?
In figures 3 and 4 are snapshots for NVMe QVIP test plan covering some of the coverpoints/crosses explained in Section 3 of this article.
Figure 3 - NVMe QVIP Test Plan (Controller Registers):
Figure 4 - NVMe QVIP Test Plan (Command Covergroup):
NVMe QVIP includes a host agent and controller agent which are instantiated in the environment class; env_config defines the various host and controller agent configurations. During the build_phase of nvme_ test class we create the nvme environment and nvme environment configurations.
In the build phase of nvme_env class:
The same can be done for the controller agent. Here “nvme_cfg” is nvme_env_config class handle which contains configurations for host and controller agents.
Now, in the build_phase of nvme_test class, we create the top level environment class.
To set any configuration for host side:
nvme_cfg.host.(config to be set)
Similarly for controller it is done as:
nvme_cfg.controller.(config to be set)
All the required configurations are provided by the NVMe QVIP nvme_vip_config class comprising of various configurations for host and controller sides which can be used by the user for their DUT settings.
Figure 5 - NVMe QVIP Environment:
In the figure below, the monitor has the analysis port which broadcasts transactions and is received by the coverage component containing the analysis export for verification.
NVMe QVIP offers a monitor component which picks up the bus transactions and converts them into nvme_txn packets, which are in turn broadcast and received by various components, like the coverage collector and scoreboard.
Figure 6 - Monitor and Coverage Collector Connections Using Analysis Ports and Exports:
NVMe QVIP Configurations for Coverage
NVMe QVIP consists of a coverage collector class which receives the NVMe transactions as broadcast by the monitor and samples the coverpoints and crosses accordingly. Now in an NVMe subsystem there can be multiple controllers with different controller IDs; therefore coverage is implemented for all of the controllers. QVIP provides a configuration for coverage for implementing coverage for the host side as well as the controller side. QVIP also provides configurations to enable specific covergroups for specific controllers.
Coverage is calculated for the host side as well as the controller side.
To enable coverage for host side:
nvme_cfg.host. (coverage_enable[for particular controller]) = 1;
For controller side:
nvme_cfg.controller. (coverage_enable[for particular controller]) = 1;
Suppose a controller doesn’t support the virtualization management feature (optional), then the QVIP provides the user support to disable all coverpoint/crosses related to virtualization management (VM) and excludes them from the coverage calculation.
This can be done in build_phase for nvme_test class:
QVIP comprises of a coverage collector class which is connected to nvme_monitor component and contains sampling logic for all the coverpoints/crosses which are part of a separate covergroups class.
QVIP runs directed tests for verifying the NVMe design and merge the generated UCDB files to achieve 100% coverage, as shown in figures 7 and 8.
Figure 7 - NVMe QVIP Flow for Functional Coverage:
Figure 8 - Questa® GUI Showing 100% Coverage:
This article gives insight into the basics of the NVMe protocol; things which must be verified as part of a test plan. It also explores how QVIP is used for the verification of NVMe using functional coverage and what things are covered as part of the NVMe QVIP test plan.
Back to Top