Verification Academy

Search form

My Account Menu

  • Register
  • Log In
  • Topics
  • Courses
  • Forums
  • Patterns Library
  • Cookbooks
  • Events
  • More
  • All Topics
    The Verification Academy offers users multiple entry points to find the information they need. One of these entry points is through Topic collections. These topics are industry standards that all design and verification engineers should recognize. While we continue to add new topics, users are encourage to further refine collection information to meet their specific interests.
    • Languages & Standards

      • Portable Test and Stimulus
      • Functional Safety
      • Design & Verification Languages
    • Methodologies

      • UVM - Universal Verification Methodology
      • UVM Framework
      • UVM Connect
      • FPGA Verification
      • Coverage
    • Techniques & Tools

      • Verification IP
      • Simulation-Based Techniques
      • Planning, Measurement, and Analysis
      • Formal-Based Techniques
      • Debug
      • Clock-Domain Crossing
      • Acceleration
  • All Courses
    The Verification Academy is organized into a collection of free online courses, focusing on various key aspects of advanced functional verification. Each course consists of multiple sessions—allowing the participant to pick and choose specific topics of interest, as well as revisit any specific topics for future reference. After completing a specific course, the participant should be armed with enough knowledge to then understand the necessary steps required for maturing their own organization’s skills and infrastructure on the specific topic of interest. The Verification Academy will provide you with a unique opportunity to develop an understanding of how to mature your organization’s processes so that you can then reap the benefits that advanced functional verification offers.
    • Universal Verification Methodology (UVM)

      • Advanced UVM
      • Basic UVM
      • Introduction to UVM
      • UVM Connect
      • UVM Debug
      • UVMF - One Bite at a Time
    • Featured Courses

      • Introduction to ISO 26262
      • Introduction to DO-254
      • Clock-Domain Crossing Verification
      • Portable Stimulus Basics
      • Power Aware CDC Verification
      • Power Aware Verification
      • SystemVerilog OOP for UVM Verification
    • Additional Courses

      • Assertion-Based Verification
      • An Introduction to Unit Testing with SVUnit
      • Evolving FPGA Verification Capabilities
      • Metrics in SoC Verification
      • SystemVerilog Testbench Acceleration
      • Testbench Co-Emulation: SystemC & TLM-2.0
      • Verification Planning and Management
      • VHDL-2008 Why It Matters
    • Formal-Based Techniques

      • Formal Assertion-Based Verification
      • Formal-Based Technology: Automatic Formal Solutions
      • Formal Coverage
      • Getting Started with Formal-Based Technology
      • Handling Inconclusive Assertions in Formal Verification
      • Sequential Logic Equivalence Checking
    • Analog/Mixed Signal

      • AMS Design Configuration Schemes
      • Improve AMS Verification Performance
      • Improve AMS Verification Quality
  • All Forum Topics
    The Verification Community is eager to answer your UVM, SystemVerilog and Coverage related questions. We encourage you to take an active role in the Forums by answering and commenting to any questions that you are able to.
    • UVM Forum

      • Active Questions
      • Solutions
      • Replies
      • No Replies
      • Search
      • UVM Forum
    • SystemVerilog Forum

      • Active Questions
      • Solutions
      • Replies
      • No Replies
      • Search
      • SystemVerilog Forum
    • Coverage Forum

      • Active Questions
      • Solutions
      • Replies
      • No Replies
      • Search
      • Coverage Forum
    • Additional Forums

      • Announcements
      • Downloads
      • OVM Forum
  • Patterns Library
    The Verification Academy Patterns Library contains a collection of solutions to many of today's verification problems. The patterns contained in the library span across the entire domain of verification (i.e., from specification to methodology to implementation—and across multiple verification engines such as formal, simulation, and emulation).
    • Implementation Patterns

      • Environment Patterns
      • Stimulus Patterns
      • Analysis Patterns
      • All Implementation Patterns
    • Specification Patterns

      • Occurrence Property Patterns
      • Order Property Patterns
      • All Specification Patterns
    • Pattern Resources

      • Start Here - Patterns Library Overview
      • Whitepaper - Taking Reuse to the Next Level
      • Verification Horizons - The Verification Academy Patterns Library
      • Contribute a Pattern to the Library
  • All Cookbooks
    Find all the methodology you need in this comprehensive and vast collection. The UVM and Coverage Cookbooks contain dozens of informative, executable articles covering all aspects of UVM and Coverage.
    • UVM Cookbook

      • UVM Basics
      • Testbench Architecture
      • DUT-Testbench Connections
      • Configuring a Test Environment
      • Analysis Components & Techniques
      • End Of Test Mechanisms
      • Sequences
      • The UVM Messaging System
      • Other Stimulus Techniques
      • Register Abstraction Layer
      • Testbench Acceleration through Co-Emulation
      • Debug of SV and UVM
      • UVM Connect - SV-SystemC interoperability
      • UVM Versions and Compatibility
      • UVM Cookbook
    • Coding Guidelines & Deployment

      • Code Examples
      • UVM Verification Component
      • Package/Organization
      • Questa/Compiling UVM
      • SystemVerilog Guidelines
      • SystemVerilog Performance Guidelines
      • UVM Guidelines
      • UVM Performance Guidelines
    • Coverage Cookbook

      • Introduction
      • What is Coverage?
      • Kinds of Coverage
      • Specification to Testplan
      • Testplan to Functional Coverage
      • Bus Protocol Coverage
      • Block Level Coverage
      • Datapath Coverage
      • SoC Coverage Example
      • Requirements Writing Guidelines
      • Coverage Cookbook
  • All Events
    No one argues that the challenges of verification are growing exponentially. What is needed to meet these challenges are tools, methodologies and processes that can help you transform your verification environment. These recorded seminars from Verification Academy trainers and users provide examples for adoption of new technologies and how to evolve your verification process.
    • Upcoming & Featured Events

      • CDC+RDC Analysis - 4/20
      • Low Power Verification - 4/29
      • Fault Campaign for Mixed-Signal - 5/4
      • User2User - 5/26
      • Webinar Calendar
    • On-Demand Webinars

      • Basic Abstraction Techniques
      • Safety Analysis Techniques
      • QVIP Workflow and Debug for PCIe
      • Writing a Proxy-driven Testbench
      • Achieving High Defect Coverage
      • Visualizer Features
      • Questa Static and Formal Apps
      • All On-Demand Webinars
    • Recording Archive

      • Siemens EDA 2021 Functional Verification Webinar Series
      • Improving Your SystemVerilog & UVM Skills
      • Should I Kill My Formal Run?
      • Visualizer Debug Environment
      • Industry Data & Surveys
      • All Recordings
    • Conferences

      • DVCon 2021
      • DVCon 2020
      • DAC 2019
      • All Conferences
    • Mentor Learning Center

      • SystemVerilog Fundamentals
      • SystemVerilog UVM
      • View all Learning Paths
  • About Verification Academy
    The Verification Academy will provide you with a unique opportunity to develop an understanding of how to mature your organization's processes so that you can then reap the benefits that advanced functional verification offers.
    • Blog & News

      • Verification Horizons Blog
      • Academy News
      • Academy Newsletter
      • Technical Resources
    • Verification Horizons Publication

      • Verification Horizons - March 2021
      • Verification Horizons - November 2020
      • Verification Horizons - July 2020
      • Issue Archive
    • About Us

      • Verification Academy Overview
      • Subject Matter Experts
      • Contact Us
    • Training

      • Questa Basic
      • Questa Advanced
      • Mastering Questa
  • Home
  • Verification Horizons
  • June 2015
  • Intelligent Testbench Automation with UVM and Questa®

Intelligent Testbench Automation with UVM and Questa®

Verification Horizons - Tom Fitzpatrick, Editor

Intelligent Testbench Automation with UVM and Questa® by Marcela Simkova and Neil Hand, VP of Marketing and Business Development Codasip Ltd.

This article describes an automated approach to improve design coverage by utilizing genetic algorithms added to standard UVM verification environments running in Questa® from Mentor Graphics®. To demonstrate the effectiveness of the approach, the article will utilize real-world data from the verification of a 32-bit ASIP processor from Codasip®.

INTRODUCTION

Application-specific instruction-set processors (ASIP) have become an integral part of embedded systems, as they can be optimized for high performance, small area, and low power consumption by changing the processor’s instruction- set to complement the class of target applications. This ability makes them an ideal fit for many applications — including the Internet of Things (IoT) and medical devices — that require advanced data processing with extremely low power.

The ASIP used in this example was modeled using Codasip® Studio, a highly integrated development environment that allows rapid ASIP prototyping and development. Using Codasip Studio, the instruction set of an ASIP — the instruction-accurate model (IA) — and the behavior of that ASIP at the hardware level — the cycle-accurate model (CA) — are described in a processor description language called CodALTM. Using this description, everything needed for implementation and integration of the ASIP is automatically generated; including the HDL representation of the ASIP (in Verilog, VHDL, or SystemC), a complete ASIP toolchain based on LLVM/GNU (compiler, simulator, debugger, profiler, assembler, and libraries), a virtual prototype (SystemC and QEMU), and the UVM based verification environment.

The power and flexibility of ASIPs, however, raises some important verification issues. One concern is the verification of the generated HDL description of the processor with respect to the reference model (e.g., the pipeline behavior, communication with a memory, etc.). Another is the verification of the ability of the ASIP to execute software applications (sequences of instructions) correctly. In practice, verification is achieved by evaluating a large amount of applications (programs) in simulation. These applications originate from various benchmarks/test suites and can be manually written or automatically generated.

UVM GENERATION WITH CODASIP STUDIO

Codasip Studio automates the generation of UVM environments for execution in the Questa® simulator from Mentor Graphics® . In this environment, the HDL representations of either individual ASIPs or complex platforms consisting of several ASIPs, buses, memories, and other IP components act as the design under verification (DUV). The reference model, which is a very important part in the UVM architecture, is automatically generated from the high-level IA model of an ASIP and from C++ models of external IP components. A generator of random applications for the target ASIP is included in the generated ASIP toolchain and will be used to generate stimuli for ASIP verification. For illustration, see Figure 1.

Figure 1. Automated generation of the HDL repre- sentation of an ASIP and the UVM environment with the reference model from Codasip Studio.
Figure 1. Automated generation of the HDL representation of an ASIP and the UVM environment with the reference model from Codasip Studio.


An important feature of the generated UVM environment is the optimization algorithm that runs in the background and adjusts the constraints of the random application generator in order to automate and speed up coverage closure.

This optimization reduces the effort needed to prepare comprehensive verification stimuli and improves verification productivity. The remainder of this article will look at how this UVM environment was structured and how it has been used for real-world ASIP verification.

THE EXPERIMENTAL ASIP

Codix-RISCTM is a 32-bit processor with six pipeline stages. It contains 32 x 32b general purpose registers with up to three read and one write ports by default.

Hazards are handled by the generated LLVM compiler, although hardware hazard management is an option. Codix-RISC is suitable for embedded signal, video, and wireless processing applications that include some application specific code, such as video codecs, malware detection, security algorithms, and any other computationally demanding low-power tasks. In the standard verification of Codix-RISC, different test applications are evaluated. Initially, benchmark applications and language test suites are used; then applications from the random application generator are applied.

PROPOSED UVM OPTIMIZATION TO IMPROVE COVERAGE

Coverage metrics tracked for Codix-RISC in the standard UVM-based environment are: code coverage (statement, branch, expression, FSM), functional coverage (requests, status, and responses on the bus interface), and instruction coverage (the complete instruction set and sequences of instructions).

In the previously described standard verification, no feedback is provided to the generator about the achieved coverage. In the random approach, the generator just produces random applications; therefore, to achieve reasonable coverage, a considerable number of applications must be applied to the ASIP. To optimize this approach, we incorporated a genetic algorithm (GA) as the main optimization tool, which provides feedback about achieved coverage and modifies the constraints of the generator. In particular, the additional constraints restrict the size of applications (100–1000 instructions) and define the probability that each instruction from the instruction-set will be generated. The generator works with its original basic set of constraints, plus the probability and size constraints, the values of which are modified by the GA.

GA employs a population of candidate solutions that evolve through several generations. The quality of candidate solutions is determined by a fitness function. According to the fitness function, the best solutions are selected and serve as parents for the next generation. New candidate solutions are created by genetic operator mutation and crossover. If the algorithm and its parameters are tuned well, the average fitness of the population improves over time. This means the algorithm is spending effort in exploring profitable parts of the search space. At the same time, genetic operators ensure diversity, so the algorithm is resilient to the problem of local optimum. For coverage- driven verification, GA serves unconventionally as an optimizer that runs in the background of the verification process. This means that in contrast to the typical application of GA, it is not intended only for searching for good candidate solutions. Moreover, as profitable values of GA parameters are found, additional tuning is no longer required. These parameters have been tested during verification of several ASIPs, as well as other IP components, and resulted in a stable level of optimization.

The GA-driven approach is provided as an extension of the basic functional verification environment prepared according to the UVM and follows the principle of object oriented programming (OOP). Our aim was to integrate the GA components effectively so that interference to the standard architecture of UVM is minimal. Figure 2 highlights the SystemVerilog components/classes that are added to the standard UVM environment.

Figure 2. The UVM environment with GA components.
Figure 2. The UVM environment with GA components.


The GA component represents the core of the algorithm. It produces chromosomes (coding representations of candidate solutions) with encoded constraints that are passed through the Chromosome Sequencer and then propagated to the random application generator. The generator then generates one application per chromosome. For the Codix-RISC processor, generated applications are directly loaded to the program memory of the processor using the Application Loader component, and the verification in simulation is started immediately.

The following is an example of simplified UVM source code of the CodixRiscChromosome class, and of the run_phase of the CodixRiscTest class.

class CodixRiscChromosome;
  real fitness; // Fitness value of chromosome
  rand bit[6:0] chromosome_parts[]; //
               Constraints for the generator
 
  function void crossover( inout CodixRisc-
                        Chromosome chrom );
   // Temporary chromosome
   CodixRiscChromosome tmpChrom = new();
 
   // Position of crossover
   int main_pos = $urandom_range
                 (CONSTRAINTS_NUMBER);
 
   tmpChrom = chrom.clone();
   chrom.chromosome_parts[main_pos] =
     this.chromosome_parts[main_pos];
   this.chromosome_parts[main_pos] =
     tmpChrom.chromosome_parts[main_pos];
 
 endfunction: crossover
 
 function CodixRiscChromosome mutate
         ( int unsigned maxMutations );
   // Position of mutation
   int main_pos;
 
   // Number of mutations
   int mutationCount = $urandom_range
                      (maxMutations);
 
 for (int i=0; i < mutationCount; i++) begin
   // Position of crossover
   main_pos = $urandom_range
             (CONSTRAINTS_NUMBER);
   // Make bit flips at the position
   bitflips(this.chromosome_parts[main_pos]);
  end
 endfunction : mutate
 
endclass: CodixRiscChromosome;
 
class CodixRiscTest extends CodixRiscTestBase;
  // registration of component tools
  `uvm_component_utils( codix_risc_platform_
                       ca_t_test )
 
...
 
 task run_phase( uvm_phase phase );
  CodixRiscChromosome best_chromosome;
  Population p, new_p;
  int counter = 0;
 
  p = createOrLoadInitialPopulation();
 
  // Evaluate population in simulation
  evaluatePopulation(p);
 
  // Get best chromosome
  best_chromosome = getBestChromosome(p);
 
  while (counter < NUMBER_OF_GENERATIONS)
        begin
   // create new population
   new_p = new();
 
   // check elitism
   if (ELITISM) new_p[0] = best_chromosome;
 
   // select parents and create new chromosomes
      by crossover, mutation
   new_p = selectAndReplace(p);
 
   // Evaluate new chromosomes in simulation
   evaluatePopulation(new_p);
 
   // Get best chromosome
   best_chromosome = getBestChromosome
                     (new_p);
 
   // create final population
   p = createNewPopulation(p, new_p);
 
   counter ++;
  end
 endtask: run_phase
 
endclass: CodixRiscTest;
 

We compared the effectiveness of the GA optimization to two standard approaches. In the first approach, the benchmark applications are used for verification of the ASIP. In the second approach, the random application generator is used. The proposed GA-driven approach also uses the random application generator but adds additional constraints to the generator encoded in the chromosome. The graph in Figure 3 demonstrates the results of these three approaches. It compares average values from 20 different measures for each approach (using various seeds). The x-axis represents the number of evaluated applications on the Codix-RISC processor, and the y-axis shows the achieved level of total coverage.

The average level of coverage achieved by 1000 benchmark applications was 88 percent. The average level of coverage achieved by 1000 random applications was 97.3 percent. In the GA-driven approach, the random generator produced random applications for 20, 50, and 100 generations (the 100 generations scenario is captured in the graph). The size of the population was always the same: ten chromosomes. For the test with 20 generations, the average level of coverage was 97.78 percent. For the test with 50 generations, the average level of coverage was 98.72 percent. For the test with 100 generations, the average level of coverage was 98.89 percent.

The results show that for all tests utilizing GA optimization, we were able to achieve better coverage than with standard approaches. It is important to mention another significant advantage of this approach. When selecting applications generated for the best chromosome in each generation, the result is a set of very few applications with very good coverage. So for example, for the GA-optimization running 20 generations we can get the best 20 applications from every generation that are able to achieve 98.1 percent coverage. These applications can be used for very efficient regression testing.

Figure 3: The comparison of standard verification approaches and the proposed GA-driven approach.
 The comparison of standard verification approaches and the proposed GA-driven approach.


The computational burden of the GA optimization was as follows. Evaluating one application in simulation took an average of 12.626 seconds, generating one application around one second and preparing a new population in 0.095 second. Experiments ran on a 3.33 GHz Intel® Core TM i5 CPU with 8 GB of RAM using the Questa simulator.

SUMMARY

We have shown that GA easily integrates into a Questa UVM verification environment, is computationally efficient, and significantly reduces the time and effort required to achieve needed coverage for ASIP based designs.

Back to Top

Table of Contents

Verification Horizons Articles:

  • Welcome to our once-again-super-sized DAC summer edition!

  • Verifying Airborne Electronics Hardware: Automating the Capture of Assertion Verification Results for DO-254

  • DO-254 Testing of High Speed FPGA Interfaces

  • Formal and Assertion-Based Verification of MBIST MCPs

  • Starting Formal Right from Formal Test Planning

  • Reuse MATLAB® Functions and Simulink® Models in UVM Environments with Automatic SystemVerilog DPI Component Generation

  • Intelligent Testbench Automation with UVM and Questa®

  • Unit Testing Your Way to a Reliable Testbench

  • Hardware Emulation: Three Decades of Evolution – Part II

  • Accelerating RTL Simulation Techniques

  • Emulation Based Approach to ISO 26262 Compliant Processors Design

  • Resolving the Limitations of a Traditional VIP for PHY Verification

Siemens Digital Industries Software

Siemens Digital Industries Software

##TodayMeetsTomorrow

Solutions

  • Cloud
  • Mendix
  • Siemens EDA
  • MindSphere
  • Siemens PLM
  • View all portfolio

Explore

  • Digital Journeys
  • Community
  • Blog
  • Online Store

Siemens

  • About Us
  • Careers
  • Events
  • News and Press
  • Newsletter
  • Customer Stories

Contact Us

USA:

phone-office +1 800 547 3000

See our Worldwide Directory

  • Contact Us
  • Support Center
  • Give us Feedback
©2021 Siemens Digital Industries Software. All Rights Reserved.
Terms of Use Privacy Cookie Policy