Verification Academy

Search form

My Account Menu

  • Register
  • Log In
  • Topics
  • Courses
  • Forums
  • Patterns Library
  • Cookbooks
  • Events
  • More
  • All Topics
    The Verification Academy offers users multiple entry points to find the information they need. One of these entry points is through Topic collections. These topics are industry standards that all design and verification engineers should recognize. While we continue to add new topics, users are encourage to further refine collection information to meet their specific interests.
    • Languages & Standards

      • Portable Test and Stimulus
      • Functional Safety
      • Design & Verification Languages
    • Methodologies

      • UVM - Universal Verification Methodology
      • UVM Framework
      • UVM Connect
      • FPGA Verification
      • Coverage
    • Techniques & Tools

      • Verification IP
      • Simulation-Based Techniques
      • Planning, Measurement, and Analysis
      • Formal-Based Techniques
      • Debug
      • Clock-Domain Crossing
      • Acceleration
  • All Courses
    The Verification Academy is organized into a collection of free online courses, focusing on various key aspects of advanced functional verification. Each course consists of multiple sessions—allowing the participant to pick and choose specific topics of interest, as well as revisit any specific topics for future reference. After completing a specific course, the participant should be armed with enough knowledge to then understand the necessary steps required for maturing their own organization’s skills and infrastructure on the specific topic of interest. The Verification Academy will provide you with a unique opportunity to develop an understanding of how to mature your organization’s processes so that you can then reap the benefits that advanced functional verification offers.
    • Universal Verification Methodology (UVM)

      • Advanced UVM
      • Basic UVM
      • Introduction to UVM
      • UVM Connect
      • UVM Debug
      • UVMF - One Bite at a Time
    • Featured Courses

      • Introduction to ISO 26262
      • Introduction to DO-254
      • Clock-Domain Crossing Verification
      • Portable Stimulus Basics
      • Power Aware CDC Verification
      • Power Aware Verification
      • SystemVerilog OOP for UVM Verification
    • Additional Courses

      • Assertion-Based Verification
      • An Introduction to Unit Testing with SVUnit
      • Evolving FPGA Verification Capabilities
      • Metrics in SoC Verification
      • SystemVerilog Testbench Acceleration
      • Testbench Co-Emulation: SystemC & TLM-2.0
      • Verification Planning and Management
      • VHDL-2008 Why It Matters
    • Formal-Based Techniques

      • Formal Assertion-Based Verification
      • Formal-Based Technology: Automatic Formal Solutions
      • Formal Coverage
      • Getting Started with Formal-Based Technology
      • Handling Inconclusive Assertions in Formal Verification
      • Sequential Logic Equivalence Checking
    • Analog/Mixed Signal

      • AMS Design Configuration Schemes
      • Improve AMS Verification Performance
      • Improve AMS Verification Quality
  • All Forum Topics
    The Verification Community is eager to answer your UVM, SystemVerilog and Coverage related questions. We encourage you to take an active role in the Forums by answering and commenting to any questions that you are able to.
    • UVM Forum

      • Active Questions
      • Solutions
      • Replies
      • No Replies
      • Search
      • UVM Forum
    • SystemVerilog Forum

      • Active Questions
      • Solutions
      • Replies
      • No Replies
      • Search
      • SystemVerilog Forum
    • Coverage Forum

      • Active Questions
      • Solutions
      • Replies
      • No Replies
      • Search
      • Coverage Forum
    • Additional Forums

      • Announcements
      • Downloads
      • OVM Forum
  • Patterns Library
    The Verification Academy Patterns Library contains a collection of solutions to many of today's verification problems. The patterns contained in the library span across the entire domain of verification (i.e., from specification to methodology to implementation—and across multiple verification engines such as formal, simulation, and emulation).
    • Implementation Patterns

      • Environment Patterns
      • Stimulus Patterns
      • Analysis Patterns
      • All Implementation Patterns
    • Specification Patterns

      • Occurrence Property Patterns
      • Order Property Patterns
      • All Specification Patterns
    • Pattern Resources

      • Start Here - Patterns Library Overview
      • Whitepaper - Taking Reuse to the Next Level
      • Verification Horizons - The Verification Academy Patterns Library
      • Contribute a Pattern to the Library
  • All Cookbooks
    Find all the methodology you need in this comprehensive and vast collection. The UVM and Coverage Cookbooks contain dozens of informative, executable articles covering all aspects of UVM and Coverage.
    • UVM Cookbook

      • UVM Basics
      • Testbench Architecture
      • DUT-Testbench Connections
      • Configuring a Test Environment
      • Analysis Components & Techniques
      • End Of Test Mechanisms
      • Sequences
      • The UVM Messaging System
      • Other Stimulus Techniques
      • Register Abstraction Layer
      • Testbench Acceleration through Co-Emulation
      • Debug of SV and UVM
      • UVM Connect - SV-SystemC interoperability
      • UVM Versions and Compatibility
      • UVM Cookbook
    • Coding Guidelines & Deployment

      • Code Examples
      • UVM Verification Component
      • Package/Organization
      • Questa/Compiling UVM
      • SystemVerilog Guidelines
      • SystemVerilog Performance Guidelines
      • UVM Guidelines
      • UVM Performance Guidelines
    • Coverage Cookbook

      • Introduction
      • What is Coverage?
      • Kinds of Coverage
      • Specification to Testplan
      • Testplan to Functional Coverage
      • Bus Protocol Coverage
      • Block Level Coverage
      • Datapath Coverage
      • SoC Coverage Example
      • Requirements Writing Guidelines
      • Coverage Cookbook
  • All Events
    No one argues that the challenges of verification are growing exponentially. What is needed to meet these challenges are tools, methodologies and processes that can help you transform your verification environment. These recorded seminars from Verification Academy trainers and users provide examples for adoption of new technologies and how to evolve your verification process.
    • Upcoming & Featured Events

      • Low Power Verification - 4/29
      • Fault Campaign for Mixed-Signal - 5/4
      • User2User - 5/26
      • Webinar Calendar
    • On-Demand Webinars

      • CDC+RDC Analysis
      • Basic Abstraction Techniques
      • Safety Analysis Techniques
      • QVIP Workflow and Debug for PCIe
      • Writing a Proxy-driven Testbench
      • Achieving High Defect Coverage
      • Visualizer Features
      • All On-Demand Webinars
    • Recording Archive

      • Siemens EDA 2021 Functional Verification Webinar Series
      • Improving Your SystemVerilog & UVM Skills
      • Should I Kill My Formal Run?
      • Visualizer Debug Environment
      • Industry Data & Surveys
      • All Recordings
    • Conferences

      • DVCon 2021
      • DVCon 2020
      • DAC 2019
      • All Conferences
    • Mentor Learning Center

      • SystemVerilog Fundamentals
      • SystemVerilog UVM
      • View all Learning Paths
  • About Verification Academy
    The Verification Academy will provide you with a unique opportunity to develop an understanding of how to mature your organization's processes so that you can then reap the benefits that advanced functional verification offers.
    • Blog & News

      • Verification Horizons Blog
      • Academy News
      • Academy Newsletter
      • Technical Resources
    • Verification Horizons Publication

      • Verification Horizons - March 2021
      • Verification Horizons - November 2020
      • Verification Horizons - July 2020
      • Issue Archive
    • About Us

      • Verification Academy Overview
      • Subject Matter Experts
      • Contact Us
    • Training

      • Questa Basic
      • Questa Advanced
      • Mastering Questa
  • Home
  • Verification Horizons
  • June 2017
  • Verification Planning with Questa® Verification Management

Verification Planning with Questa® Verification Management

Verification Horizons - Tom Fitzpatrick, Editor

Verification Planning with Questa® Verification Management by Kishan Kalavadiya and Bhavinkumar Rajubhai Patel - eInfochips

Verification of complex SoC (System on Chip) requires tracking of all low level data (i.e. regression results, functional and code coverage). Usually, verification engineers do this type of tracking manually or using some automation through scripting. Manual efforts in order to get above information while verifying complex SoC may lead us towards the delay in project execution. A verification planning tool can help to reduce such manual efforts and make the tracking process more efficient. Mentor, A Siemens Business has such a Verification Planning tool for QuestaSim within their Verification Management tool suite known as "Questa® Testplan Tracking". This article contains detailed steps to use this tracking process along with key features which can reduce the time in verification cycle to track the verification progress.

INTRODUCTION

Mentor's Verification Testplan Tracking facilitates the verification process management across all relevant aspects. Without such automation, tracking of verification progress requires a lot of manual effort in order to get information like coverage and regression status in a common place. Functional and code coverage numbers are key parameters in verification and without it verification cannot be closed. Regression analysis (individual test status) is also an important aspect of tracking the verification progress. Questa® Verification Management automates the tracking of verification progress by offering the analysis and optimization facilities followed by additional features like trend analysis, back annotation of results, filtration of specific data and other features. It provides the top level summary of all verification aspects, which provides real time visibility into the project to all parties, i.e. designers, verification engineers, project managers, etc. Due to this visibility, it becomes easy to manage risk factors and efficiency so project execution can be improved.

MOTIVATION FOR QUESTA® TESTPLAN DEVELOPMENT

Functional verification has been described as a major challenge in SoC designs with proper visibility into the verification process as a major factor contributing to this challenge. This lack of visibility impacts design quality, schedule predictability and cost.

A testplan is a document which captures the important features of a design and details regarding how they will be verified. Questa testplan is such a plan which can be linked directly to the coverage database and results can be annotated in that plan itself. By putting such a plan in place that captures list of verification intent, one can organize the tracking of verification progress better. While verifying the complex SoC (System on Chip), it may be possible that all the features we have listed in the Questa Testplan do not have equal priority. For that case, one can add separate user defined attributes in the testplan to provide priority for individual features. Verification engineers can focus their coverage closure efforts on the critical features that have been identified as a whole followed by important features and then by the "nice to have" features.

BASIC FLOW FOR QUESTA® VM VERIFICATION PLANNING

Questa Testplan includes feature wise test scenarios, functionality covered by that feature (functional coverage), assertions, code coverage and such more in XML format. After preparing such Questa Testplan, conversion of such plan in UCDB (Unified Coverage Database) format is required using xml2ucdb command. After running the regression, separate coverage databases will be available for individual test scenarios. At the time of merging of these coverage databases, Questa Testplan in UCDB format needs to be merged so that the results can be linked with the data available in Questa Testplan. One can back annotate the results in XML format of Questa Testplan and the same results will be available in HTML report or in GUI as well. Creation of the Questa Testplan required certain steps and syntax, which is explained in the upcoming section.

Figure 1. Basic Flow Diagram


DETAILED STEPS FOR QUESTA® TESTPLAN DEVELOPMENT

In many cases, the features will be verified in simulation and recorded as verified using coverage analysis. For Questa Testplan which facilitates back annotation of information, linking between the testplan and simulations is required. For that, certain document format must be followed. The format for Questa Testplan is described below with detailed steps.

Step: 1. Install Questa add-in in Microsoft® Excel

By default, Microsoft Excel does not contain Questa option in toolbar. So the user needs to install Questa add-in which is open source and freely available. After installing this add-in, it will be available in the toolbar as shown in figure 2.

Figure 2. Questa add-in in Microsoft Office Toolbar


Step: 2. Prepare the Questa Testplan

For creating the Questa Testplan, select create testplan option from Questa VM option as shown in figure 3.

Figure 3. Questa® VM options and Testplan Details


Questa's Testplan Tracking requires four distinct pieces of information for each requirement captured. They are Section, Title, Link and Type. Link is one of the attributes used while developing the Questa Testplan, which is required to link plan information to the actual results. The additional information for each requirement includes a Description, a Weight and a Goal. While these additional fields are not required, they are used the majority of the time. Questa's Testplan Tracking also has the flexibility to allow user defined fields. If we look at the typically used fields in a spreadsheet format, each field is represented by a column in the spreadsheet as shown in figure 4.

Figure 4. Sample Questa Testplan Format


After preparing the Questa Testplan, it can be saved as XML or can be directly exported to XML from Questa VM options as shown in figure 3 above.

User can track different information by specifying the different options in Type field. Description for options available for Type field is as shown in table 1, below.

Table 1. Coverage Construct Type Field options


According to the requirement, user can set several rules as per the option available in Add Rule option. In figure 5, rule is applied to get the recursive toggle and statement coverage for some particular module, as shown below.

Figure 5. Add Rule option in Questa Testplan


Step: 3. XML to UCDB Conversion of Questa Testplan

Convert this XML Questa Testplan in UCDB using the command below:

xml2ucdb -format Excel verification_plan.xml -ucdbfilename verification_plan.ucdb

Step: 4. Regression and Coverage Database generation

After preparing the Testplan, next step is to develop test cases to verify the features listed in plan. After running the regression of those test cases, command for merge all separate coverage databases is as shown:

vcover merge merged_cov.ucdb <test1.ucdb> <test2.ucdb>…<testn.ucdb> verification_plan.ucdb

We can back annotate the results in existing XML format Questa Testplan using option below:

Questa VM > coverage data > annotate

Step: 5. HTML Report generation

Command for the HTML report generation from merged coverage database is as shown below:

vcover report -html -htmldir covreport merged_cov.ucdb -details -testhitdata

VERIFICATION RESULTS ANALYSIS

Questa's Testplan Tracking facilitates with multiple formats for analysis of the results hierarchically so that it will become easy to address the failures identified in regression and track the coverage holes to complete the verification process. Find sample reports below for analysis in different format.

Figure 6 and figure 7 contain the top level summary of Questa Testplan in GUI and HTML format respectively.

Figure 6. Verification Management Tracker in GUI


Figure 7. Questa Testplan in HTML


Annotated Questa Testplan in XML format in figure 8 contains the same information which is shown in figure 6 and figure 7. Figure 8 (on the following page) shows the syntax for different Link options as per the requirements.

Figure 8. Annotated Questa Testplan in XML


Asterisk (*) is permissible in case of the same naming convention. Here in the above annotated Questa Testplan, section 2.5 containing 4 random test cases, out of which, 3 are passing and 1 is failing. To link these test cases with merged database, ' * ' is being used in Link attribute. Here, pass/fail status of these test cases is in terms of coverage itself (i.e., getting 75% coverage for 4 objects means 3 test cases are passing out of 4 random cases). Similarly, we can get the results in cases of multiple iteration of the same test case.

Filtration is also one of the key features of Testplan Tracking, with which a user can filter out user-specific results from the whole report. A user can use Attribute Name, Coverage Numbers, Weight, Link Type, Instance Type and much more to filter the required data. Users need to create individual filters according to the requirements, as shown in the GUI illustrated in figure 9.

Select Tools > Filter > Setup > Create

Users can apply the filter using the option below:

Select Tools > Filter > Apply > Name_of_Filter

Example for the filtration feature is shown in figure 9.

Figure 9. Filtration in Verification Planner


TREND ANALYSIS

Trend analysis offers the ability to track the progress of the coverage over time of all the objects. For trend analysis, user needs to manage one trend UCDB after each regression. Trend UCDB is nothing but a special purpose UCDB file which contains the coverage data over the period for trend analysis.

Trend reporting offers 2-dimensional representation of coverage number over the period for individual feature or instance of DUT. Trend report can be viewed in multiple formats (i.e. HTML Report, XML Report, etc.)

Trend reports appear in the GUI using trend UCDB at Verification Management > Browser > Trend Analysis

Command for trend UCDB generation from command line is as shown below:

vcover merge -trend [-output] <trend ucdb> <ucdb inputs>

Sample trend reports and a graphical representation for the same is shown in figure 10 and figure 11 respectively.

Figure 10. Date wise trend report in VM Trender (GUI)


Figure 11. 2-Dimensional Graph in Trend Analysis


Conclusions

Verification of complex SoC projects is a complex process to manage without Verification Management. Automation of Verification Planning provides deeper visibility into the regression process to allow for throughput optimization. Questa VM's Testplan Tracking Tool enables us to analyze the regression failures and coverage holes more efficiently by providing the reports in multiple formats. Use of Testplan Tracking can reduce the manual effort which is required to update the verification documents at the time of closure and quicken the documentation in a more appropriate format.

References

Questa SIM Verification Management User Manual (Software Version 10.4c)

Back to Top

Table of Contents

Verification Horizons Articles:

  • What Does Improving Your Golf Swing Have in Common with Verification?

  • Parallel Debug: A Path to a Better Big Data Diaspora

  • Portable Stimulus Modeling in a High-Level Synthesis User's Verification Flow

  • Smoothing the Path to Software-Driven Verification with Portable Stimulus

  • Verification Planning with Questa® Verification Management

  • MIPI® CSI2 TX IP Verification Using Questa® VIPs

  • Converting Legacy USB IP to a Low Power USB IP

  • Understanding the UPF Power Domain and Domain Boundary

  • Automation and Reuse in RISC-V Verification Flow

  • Emulation – A Job Management Strategy to Maximize Use

  • RTL CDC Is No Longer Enough — How Gate-Level CDC Is Now Essential to First Pass Success

  • Formal Verification: Not Just for Control Paths

Siemens Digital Industries Software

Siemens Digital Industries Software

##TodayMeetsTomorrow

Solutions

  • Cloud
  • Mendix
  • Siemens EDA
  • MindSphere
  • Siemens PLM
  • View all portfolio

Explore

  • Digital Journeys
  • Community
  • Blog
  • Online Store

Siemens

  • About Us
  • Careers
  • Events
  • News and Press
  • Newsletter
  • Customer Stories

Contact Us

USA:

phone-office +1 800 547 3000

See our Worldwide Directory

  • Contact Us
  • Support Center
  • Give us Feedback
©2021 Siemens Digital Industries Software. All Rights Reserved.
Terms of Use Privacy Cookie Policy