In many AI (Artificial Intelligence) architectures, the design is structured in a hierarchy with a processing element consisting of one or more CPUs, plus AI accelerators or co-processor(s), plus some additional logic to connect to the SoC AI fabric. This is a critical feature to support the desired applications, and offers convenient abstraction levels to align the verification methods. Next, while the processor IP and PE have been verified, and the NoC has been verified (assuming that an existing NoC IP is used), the interaction of the RISC-V processor, PE and the NoC is unique to the design and requires verification.
Last, while the verification of a single PE is needed, verifying multiple PEs working with each other through the NoC is also needed. This is especially true in the case of designs based on RISC-V, since the Open ISA flexibility allows for optimization of each of the cores, so all the various combinations of PEs will need verification as well.
These verification challenges can be addressed within the general framework of the UVM verification methodology and tools, however, some innovation is needed, along with collaboration between processor IP vendors, EDA vendors, other tool developers and the RISC-V SoC developers. For example, several test generation and instruction stream generation tools have been developed to address RISC-V specific requirements, and new directed test suites have been developed for specific RISC-V extensions such as the vector instructions. New reference models are needed for the RISC-V processors and PEs. New metrics are needed, especially for the processor and PE verification areas, perhaps such as instruction coverage. Flows with these tools need to be robust to handle the variety of processor IP scenarios elaborated above. Plus, a robust flow is needed to ensure that a solitary “bad actor” does not insert a backdoor into the processor or SoC.
Other areas that are being looked at to address RISC-V SoC verification include using hybrid emulation-virtual platform systems for hardware software co-verification, using Portable Stimulus (PSS) for multiple PE and full chip verification, and using the nature of the AI algorithms to constrain the SoC state space.
In this article, the verification challenges for RISC-V SoCs are discussed and an overview given of potential solutions. Specific verification flows including new test and instruction stream generators, and reference models and metrics, are presented in detail including the results of using these flows on real processor IP and SoC designs. Mentor Questa® is fundamental to the RISC-V processor verification, with the RTL of the processor DUT (Device Under Test) and Imperas’ RISC-V golden reference model encapsulated in the SystemVerilog UVM testbench for lock-step comparison and testing.
COMPLIANCE IS NOT VERIFICATION
With RISC-V, as an open ISA specification1, any implementation will need to be tested against the latest RISC-V compliance suite. The objective of the compliance process is to ensure that implementations are correctly following the specifications, with the expectation that compliant devices will exhibit sufficient compatibility to leverage the emerging ecosystem for tools and software. Put more simply, compliance is confirming that the designers have understood the specifications. Since the ISA specification does not include details of microarchitecture, differences in device performance and application focus are expected and of course permitted. Since the compliance tests use expected functionality as the basis of the test suite, this incurs an overlap with some aspects of Design Verification (DV). However, the compliance suite is not exhaustive for all functionality and is focused purely with the structural specification aspects of the ISA, i.e. compliance is a subset of DV. The RISC-V Compliance Suite is developed within the RISC-V International Working Group on Compliance (“Compliance WG”), and the latest test suites are available from the RISC-V compliance GitHub repository2.
CUSTOM INSTRUCTIONS AND REFERENCE MODELS
A reference model is a key to processor-related DV tasks. This is usually an instruction accurate (IA) model of the processor, often called an Instruction Set Simulator (ISS). The Compliance WG GitHub repository includes the riscvOVPsim ISS. The riscvOVPsim simulator implements the full and complete functionality of the RISC-V Unprivileged (formally known as User) and Privilege specifications. The simulator is command-line configurable to enable/disable all current optional and processor specific options in the RISC-V specification. The simulator is developed, licensed and maintained by Imperas Software Ltd., and is fully compliant to the Open Virtual Platforms (OVP) open standard APIs. Most recently, support for the vector and bit manipulation instructions were added to the OVP RISC-V processor models. When custom instructions are added to the RISC-V processor, those instructions need to be added to the reference model. Imperas has previously developed a methodology for profiling and analysis for custom instructions, and the outline for this flow is shown in figure 1 below. The resulting model, which includes both the standard RISC-V instructions and the custom instructions, can then be used as a reference model for DV of the processor RTL.
Figure 1 - Flow for adding custom instructions to the RISC-V Processor model
There are three techniques currently being used for RISC-V processor verification: directed tests, constrained random test generation and test generation and execution.
Directed test suites are an established technique, however, what has not been previously done is the measurement of instruction coverage of these test suites. Also, with the vector extensions to the RISC-V ISA, the difficulty involved in building a comprehensive test suite is increased exponentially. For example the RISC-V vector engines have 90 different possible configurations, and nearly 500 instructions. This is obviously a complex problem.
With this article, the authors report new instruction coverage metrics, with the coverage tool included in the ISS. An example of the coverage results for the RV32I compliance test suite is shown in figure 2.
Figure 2 - Example of instruction coverage results for the RV32I compliance test suite
The authors are also developing a directed test suite (“Vector Test Suite”) for the RISC-V vector instructions. Data on this test suite will be available later in 2020.
Constrained Random Test Generation
Constrained random test generation for SoC DV is also an established technique. However, for processor DV, this needs to be an Instruction Stream Generator (ISG). Google has developed and made open source an ISG for RISC-V
Figure 3 below shows the basic flow for RISC-V processor DV using the ISG. This flow was originally developed to use a trace or signature compare methodology, but is now being evolved to support a step-and-compare methodology using the ISS encapsulated in SystemVerilog, as shown in figure 4 below.
Figure 3 - RISC-V processor DV Flow with Google open source ISG and Imperas RISC-V reference model
Figure 4 - Block diagram of SystemVerilog encapsulation of the Imperas RISC-V reference model
The encapsulation of the RISC-V reference model within SystemVerilog allows direct interaction with the testbench environment. When an issue is uncovered, a direct debug and analysis can be initiated. In an automated test environment, action can be taken to minimize the lost efficiency in continuing beyond the point of failure in a typical log compare approach. In addition, the testbench can be extended within SystemVerilog to use stimulus objects to expected response objects as both an aid for debug and a more exhaustive DV test plan. With the help and assistance of the experts at Mentor under the Vanguard program, the SystemVerilog extensions have been set-up to support the close and efficient coupling with the Imperas OVPsim simulator.
Figure 5 below shows the step-and-compare flow. This flow has been implemented for the testing of the Ibex core that was originally developed by ETH Zurich under the name “Zero-riscy” and was recently adopted by LowRISC as Ibex. Ibex implements the RISC-V RV32IMC instructions, which is the 32-bit RISC-V processor with integer (I), multiplier/divider (M) and compressed (C) instructions.
Figure 5 - Encapsulated Imperas RISC-V Reference Model in the SystemVerilog testbench for a step-and-compare DV flow
Table 1 on the following page shows the different categories of bugs found in the Ibex processor using this approach, while figure. 6, also below, shows an example of the types of bugs found.
Table 1 - Categories of bugs found using the ISG-based DV flow
Figure 6 - Examples of types of bugs found with the ISG-based flow image courtesy of Google, LLC
Test Generation and Execution
An alternative, and complementary approach to test generation for processor DV, which also can be applied to SoC DV, is the generation of tests as an executable which can be run on the RTL. In this approach, tests are randomly generated, then run on the processor reference model. The results from running the tests on the processor reference model are then combined with the random tests and used as reference test results. This flow is shown in figure 7 below.
Figure 7 - Block diagram of test generator flow, image courtesy of Valtrix Systems
PE, MULTIPLE PE AND NoC-PE VERIFICATION
The key pieces of SoC DV include verification of single PEs, verification of multiple PEs and verification of the interface between the PE and the Network on Chip (NoC) as shown in figure 8 below.
Figure 8 - Architecture Exploration for SoC Processor Arrays for AI and Machine Learning
Verification of Processing Elements (PE)
In many RISC-V based SoCs targeted at AI applications, the architecture includes Processing Elements (PEs) which have more than one RISC-V processor, plus an AI accelerator, plus some custom logic. The custom logic is typically comprised of custom instructions added to the RISC-V processors, plus additional logic for controlling the communications between processors. For these PEs, the individual processors (including custom instructions) must be verified as discussed above. However, with the integration of multiple processors into the PE, there can be different interactions that cannot be tested at the individual processor level.
For this level of integration, the PE can also be modeled using the same instruction accurate techniques that were used to model individual processors. The OVP APIs are used to build a model of the PE, and the same SystemVerilog encapsulation techniques are used to encapsulate the model of the PE, again enabling step-and-compare verification of the PE RTL.
A key piece here is the test generation. Certainly, constrained random test generation could work, however, this might spend too many cycles “re-verifying” the individual processors and not focusing on the new, unique interactions at the PE level of integration. Another possibility is to run actual software meant to execute on the real PEs. This should bring out these additional interactions.
In these AI architectures, often one PE interacts regularly with multiple neighbor PEs. It is unclear how best to verify these PE-PE interactions. Two ideas being explored now are 1) to combine the instruction accurate models with RTL simulation; and 2) to combine the instruction accurate models with hardware emulation. In the first scenario, one might have one PE represented in RTL, and the remainder of the PEs as IA models. This could enable the IA models to run the actual software, generating more interesting “stimuli” for testing the RTL PE. In the second scenario, something similar to the first is contemplated. However, in this situation the RTL blocks would be implemented in the hardware emulator. Such a hybrid IA simulation-emulation environment is shown in figure 9 below.
Figure 9 - Block diagram of hybrid IA simulation-hardware emulation environment
Verification of the interface between a processor or processor subsystem and a NoC is a well-established process. This element of RISC-V verification is raised because there has been only a limited number of RISC-V based SoCs built using the various NoCs, so DV engineers should realize that this is not the fully mature NoC interface that one receives when other processor architectures are used.
RISC-V is generating some significant attention in many market segments and applications. The freedom of the Open ISA and custom extensions together with a framework of ecosystem support provides system designers and SoC architects new options and flexibilities for optimised processor implementations. The SoC DV teams will need to accommodate processor verification in addition to addressing the flexibilities that will affect the complete SoC verification task. To maintain project schedules and tapeout deadlines verification methodologies will need to adapt and evolve to accommodate the coming wave of new SoC design complexity.
The authors wish to thank Richard Ho and Tao Liu of Google LLC for their help with the Google ISG, Valtrix for their support of STING, and the support the Imperas team received under the Mentor Questa® Vanguard Program.
Imperas is revolutionizing the development of embedded software and systems and is the leading provider of RISC-V processor models and virtual prototype solutions. Imperas, along with Open Virtual Platforms (OVP), promotes open source model availability for a spectrum of processors, IP vendors, CPU architectures, system IP and reference platform models of processors and systems ranging from simple single core bare metal platforms to full heterogeneous multi-core systems booting SMP Linux. All models are available from Imperas at www.imperas.com and the Open Virtual Platforms (OVP) website at www.ovpworld.org.
- RISC-V International ISA specification
- RISC-V International Compliance GitHub repository
- OVP (Open Virtual Platforms)
- Google ISG GitHub repository
- Ibex by LowRISC
Back to Top