Verification Academy

Search form

My Account Menu

  • Register
  • Log In
  • Topics
  • Courses
  • Forums
  • Patterns Library
  • Cookbooks
  • Events
  • More
  • All Topics
    The Verification Academy offers users multiple entry points to find the information they need. One of these entry points is through Topic collections. These topics are industry standards that all design and verification engineers should recognize. While we continue to add new topics, users are encourage to further refine collection information to meet their specific interests.
    • Languages & Standards

      • Portable Test and Stimulus
      • Functional Safety
      • Design & Verification Languages
    • Methodologies

      • UVM - Universal Verification Methodology
      • UVM Framework
      • UVM Connect
      • FPGA Verification
      • Coverage
    • Techniques & Tools

      • Verification IP
      • Simulation-Based Techniques
      • Planning, Measurement, and Analysis
      • Formal-Based Techniques
      • Debug
      • Clock-Domain Crossing
      • Acceleration
  • All Courses
    The Verification Academy is organized into a collection of free online courses, focusing on various key aspects of advanced functional verification. Each course consists of multiple sessions—allowing the participant to pick and choose specific topics of interest, as well as revisit any specific topics for future reference. After completing a specific course, the participant should be armed with enough knowledge to then understand the necessary steps required for maturing their own organization’s skills and infrastructure on the specific topic of interest. The Verification Academy will provide you with a unique opportunity to develop an understanding of how to mature your organization’s processes so that you can then reap the benefits that advanced functional verification offers.
    • Universal Verification Methodology (UVM)

      • Advanced UVM
      • Basic UVM
      • Introduction to UVM
      • UVM Connect
      • UVM Debug
      • UVMF - One Bite at a Time
    • Featured Courses

      • Introduction to ISO 26262
      • Introduction to DO-254
      • Clock-Domain Crossing Verification
      • Portable Stimulus Basics
      • Power Aware CDC Verification
      • Power Aware Verification
      • SystemVerilog OOP for UVM Verification
    • Additional Courses

      • Assertion-Based Verification
      • An Introduction to Unit Testing with SVUnit
      • Evolving FPGA Verification Capabilities
      • Metrics in SoC Verification
      • SystemVerilog Testbench Acceleration
      • Testbench Co-Emulation: SystemC & TLM-2.0
      • Verification Planning and Management
      • VHDL-2008 Why It Matters
    • Formal-Based Techniques

      • Formal Assertion-Based Verification
      • Formal-Based Technology: Automatic Formal Solutions
      • Formal Coverage
      • Getting Started with Formal-Based Technology
      • Handling Inconclusive Assertions in Formal Verification
      • Sequential Logic Equivalence Checking
    • Analog/Mixed Signal

      • AMS Design Configuration Schemes
      • Improve AMS Verification Performance
      • Improve AMS Verification Quality
  • All Forum Topics
    The Verification Community is eager to answer your UVM, SystemVerilog and Coverage related questions. We encourage you to take an active role in the Forums by answering and commenting to any questions that you are able to.
    • UVM Forum

      • Active Questions
      • Solutions
      • Replies
      • No Replies
      • Search
      • UVM Forum
    • SystemVerilog Forum

      • Active Questions
      • Solutions
      • Replies
      • No Replies
      • Search
      • SystemVerilog Forum
    • Coverage Forum

      • Active Questions
      • Solutions
      • Replies
      • No Replies
      • Search
      • Coverage Forum
    • Additional Forums

      • Announcements
      • Downloads
      • OVM Forum
  • Patterns Library
    The Verification Academy Patterns Library contains a collection of solutions to many of today's verification problems. The patterns contained in the library span across the entire domain of verification (i.e., from specification to methodology to implementation—and across multiple verification engines such as formal, simulation, and emulation).
    • Implementation Patterns

      • Environment Patterns
      • Stimulus Patterns
      • Analysis Patterns
      • All Implementation Patterns
    • Specification Patterns

      • Occurrence Property Patterns
      • Order Property Patterns
      • All Specification Patterns
    • Pattern Resources

      • Start Here - Patterns Library Overview
      • Whitepaper - Taking Reuse to the Next Level
      • Verification Horizons - The Verification Academy Patterns Library
      • Contribute a Pattern to the Library
  • All Cookbooks
    Find all the methodology you need in this comprehensive and vast collection. The UVM and Coverage Cookbooks contain dozens of informative, executable articles covering all aspects of UVM and Coverage.
    • UVM Cookbook

      • UVM Basics
      • Testbench Architecture
      • DUT-Testbench Connections
      • Configuring a Test Environment
      • Analysis Components & Techniques
      • End Of Test Mechanisms
      • Sequences
      • The UVM Messaging System
      • Other Stimulus Techniques
      • Register Abstraction Layer
      • Testbench Acceleration through Co-Emulation
      • Debug of SV and UVM
      • UVM Connect - SV-SystemC interoperability
      • UVM Versions and Compatibility
      • UVM Cookbook
    • Coding Guidelines & Deployment

      • Code Examples
      • UVM Verification Component
      • Package/Organization
      • Questa/Compiling UVM
      • SystemVerilog Guidelines
      • SystemVerilog Performance Guidelines
      • UVM Guidelines
      • UVM Performance Guidelines
    • Coverage Cookbook

      • Introduction
      • What is Coverage?
      • Kinds of Coverage
      • Specification to Testplan
      • Testplan to Functional Coverage
      • Bus Protocol Coverage
      • Block Level Coverage
      • Datapath Coverage
      • SoC Coverage Example
      • Requirements Writing Guidelines
      • Coverage Cookbook
  • All Events
    No one argues that the challenges of verification are growing exponentially. What is needed to meet these challenges are tools, methodologies and processes that can help you transform your verification environment. These recorded seminars from Verification Academy trainers and users provide examples for adoption of new technologies and how to evolve your verification process.
    • Upcoming & Featured Events

      • Creating an Optimal Safety Architecture  - February 9th
      • The ABC of Formal Verification - February 11th
      • Events Calendar
    • On Demand Seminars

      • I'm Excited About Formal...
      • Visualizer Coverage
      • Formal-based ‘X’ Verification
      • 2020 Functional Verification Study
      • All On-Demand Seminars
    • Recording Archive

      • Improving Your SystemVerilog & UVM Skills
      • Should I Kill My Formal Run?
      • Visualizer Debug Environment
      • All Recordings
    • Mentor Training Center

      • SystemVerilog for Verification
      • SystemVerilog UVM
      • UVM Framework
      • Instructor-led Training
    • Mentor Learning Center

      • SystemVerilog Fundamentals
      • SystemVerilog UVM
      • Questa Simulation Coverage Acceleration Apps with inFact
      • View all Learning Paths
  • About Verification Academy
    The Verification Academy will provide you with a unique opportunity to develop an understanding of how to mature your organization's processes so that you can then reap the benefits that advanced functional verification offers.
    • Blog & News

      • Verification Horizons Blog
      • Academy News
      • Academy Newsletter
      • Technical Resources
    • Verification Horizons Publication

      • Verification Horizons - November 2020
      • Verification Horizons - July 2020
      • Verification Horizons - March 2020
      • Issue Archive
    • About Us

      • Verification Academy Overview
      • Subject Matter Experts
      • Contact Us
    • Training

      • Questa® & ModelSim®
      • Questa® inFact
      • Functional Verification Library
  • Home
  • Verification Horizons
  • March 2016
  • Increased Efficiency with Questa® VRM and Jenkins Continuous Integration

Increased Efficiency with Questa® VRM and Jenkins Continuous Integration

Verification Horizons - Tom Fitzpatrick, Editor

by Thomas Ellis, Mentor Graphics

"Time is really the only capital that any human being has, and the only thing he can't afford to lose."

—Thomas Edison

For all the incredible technological advances to date, no one has found a way to generate additional time. Consequently, there never seems to be enough of it. Since time cannot be created, it is utterly important to ensure that it is spent as wisely as possible. Applying automation to common tasks and identifying problems earlier are just two proven ways to best utilize time during the verification process. Continuous Integration (CI) is a software practice, which is focused on doing precisely that, resulting in a more efficient use of time.

WHAT IS CONTINUOUS INTEGRATION?

The basic principle behind Continuous Integration is that the longer a branch of code is checked out, the more it begins to drift away from what is stored in the repository. The more the two diverge, the more complicated it becomes to eventually merge in changes easily, ultimately leading to what is commonly referred to as "integration hell". To avoid this, and ultimately save engineers time, CI calls for integrating regularly and often (typically daily).

Figure 1: Continuous Integration Flow.


Regular check-ins are of course, only half the equation, you need to be able to verify their changes quickly as well, otherwise many small check-ins over several days, is no different than one large check-in at weeks end. Commonly, in a Continuous Integration environment, a CI server monitors the source control for check-in's, which in turn triggers a CI process (time-based triggers are also common). This process will then build the necessary design files, and run the requisite integration tests. Once complete, the results of the tests are reported back to the user, and assuming everything passed, can now be safely committed to the repository.

By following this model, issues can be caught earlier in the development process, and can be resolved quicker as there is less variance between check-ins.

This practice has been used successfully for many years in the software industry, so much so, that it is fairly common place today. However, the idea of Continuous Integration is still fairly new in the realm of hardware verification, so it is difficult to find any published metrics on its usage as it pertains to that space specifically. However, one of the benefits of adopting a more mature technology, is you can avoid making some of the pitfalls which plagued early adopters. Since Continuous Integration technology has been used by software teams for some time, you can glean a general idea of both how widespread its usage has become, as well as what technologies have risen to the top.

ZeroTurnaround is a development company, which amongst other things, conducts an annual global survey of Java developers, and produces a report of the tools and technologies being most commonly used by the industry. In 2014, they received responses from nearly 2200 developers covering many topics, one of which was their usage of Continuous Integration Technologies. In that survey, they found that roughly 80% of (or four out of five) developers, reported using Continuous Integration in their teams. A number which itself, showed fairly significant growth, up from 68% the prior year.

Figure 2: 4 in 5 Developers Use Continuous Integration.


Another interesting aspect of the report, is the breakdown of which Continuous Integration servers were most commonly used. Far and away the most popular server was Jenkins, which was reportedly used by 70% of the developers who claimed to use CI. The second place tool was used by a mere 9% of users. So what is Jenkins, and why is it the favorite CI tool of so many users?

MEET JENKINS

Jenkins is a freely available, open-source continuous integration tool (released under the MIT license).

Figure 3: Jenkins CI.


A quick background, Jenkins was initially developed by Kohsuke Kawaguchi while he was working at Sun Microsystems in 2004. However, at the time, the project was named Hudson. After its initial release in 2005, it quickly became a favorite open-source build server. In 2010, issues began to arise between the open source community working on Hudson, and Oracle (who had since acquired Sun). Eventually requiring a vote to be called, as to whether or not to fork the project. Based on an overwhelmingly supportive community vote, Jenkins was created as a fork of Hudson. The majority of those working on, or using Hudson at the time, eventually migrated to Jenkins. Currently there are at least 127,000 installations of Jenkins (based on the anonymous usage statistics of the tool). Remember the ZeroTurnaround study? They found only 8% of users to still be using Hudson.?

Apart from being open-source, Jenkins is easy to install and highly configurable via its web interface. While Jenkins offers a lot itself, it is also highly extensible via plug-ins to the tool. At present, it boasts 1350+ plugins from 580+ contributors, to perform a myriad of different tasks, allowing for many third-party tools to leverage the power of Jenkins.

JENKINS AND VRM

On the surface, one might think that Jenkins and VRM are competitive technologies; after all, both tools can build, run and report on regressions. However, in actuality, they are truthfully complementary technologies. Furthermore, by marrying the two technologies together, you can benefit from the strengths of both, and create an extremely powerful solution for building and testing hardware designs.

While Jenkins is extremely flexible, and can run just about anything, with lots of neat bells and whistles to boot, nothing within the Jenkins core is knowledgeable about hardware verification. In the same way that VRM does not natively monitor code repositories for developer check-ins, concepts like merging SystemVerilog functional coverage, or recognizing why a UVM testbench failed are not native to Jenkins, in the way that they are at the core of VRM. What you want to do is leverage Jenkins' strengths as a build system to monitor our source repository and allow it to launch our regressions. Ultimately what it will launch though, is VRM, which will handle managing the individual verification tasks by integrating with our grid software, collecting and merging the coverage and results, etc. Once the regression is completed, Jenkins can then ask VRM to supply metrics for what was accomplished during the run, and display those results in its web dashboard.

Figure 4: Jenkins and VRM Work Together.


A REGRESSION IN JENKINS

Let's take a quick look at setting up a project to run VRM in Jenkins. On the following page is the project configuration page in Jenkins.

Here you can see the basic steps for configuring a project in Jenkins. Tasks in Jenkins are represented by builds. A build could be a complete regression, it could be the running of unit tests, or any other task you may wish Jenkins to automate.

Figure 5: Configuring a Project in Jenkins.


First you specify when to run our tests via a build trigger. A build trigger can be a period of time, a specific time, or you can even have Jenkins monitor your repository for changes, and automatically start a build for you. Jenkins will then run whatever you tell it to, which in this example, will be to launch VRM.

Finally, Jenkins will report the results of the regression run (or build). Out of the box, Jenkins will give you basic pass/fail information and some basic reporting of results, however, its lack of the metrics verification engineers are most commonly interested in, makes it feel a bit empty. To solve this shortcoming, you need one last piece to truly tie everything together neatly.

VRM JENKINS PLUG-IN

As mentioned earlier, one of the key benefits of Jenkins is that it is highly extensible through plug-ins. To get Jenkins to become more useful with respect to you running regressions with VRM, you can leverage the VRM Jenkins plug-in. You simply install the plug-in through Jenkins plug-in manager, and now Jenkins has the ability to understand code and functional coverage, determine where log files reside, monitor host utilization, and many other verification centric tasks.

To display the VRM results, and enable these features, you simply need to add what is called a post-build action (in Jenkins terms), which has Jenkins call the plug-in to make sense of the regression results.

The setup is very straightforward, you simply need to tell Jenkins where the regression ran. Additionally, you can optionally select to enable a few other features such as creating HTML reports and publishing a coverage graph to the project page. That's it! Jenkins and VRM will do the rest.

Figure 6: VRM Jenkins Plug-In.


VRM REGRESSION RESULTS IN JENKINS

One of the great features of Jenkins is its web dashboard. Now that it is using the VRM Jenkins Plug-in, you get access to a lot of great information at a glance. There is far too much to show in this short article, but here are a few examples.

The main project page has two graphs which shows you a trend of the test results, as well as the coverage results from all your past regression runs. You also get a summary table which lists the last several regressions, including information on their duration, pass/fail statistics and coverage. There are also quick links to the HTML coverage report, as well as the latest test results.

If you dig into the most recent build, you can get more detailed data on that particular regression.

Figure 7: Jenkins Project Dashboard.


Here, in addition to pass/fail and coverage results, you can also see a list of the specific tests which failed, providing a means for easy high-level inspection. Expanding a given test, will give us both the reason for the failure, as well as the standard output for the test in question.

Figure 8: Regression Results.


The plug-in leverages the vast amount of data VRM collects from the regression runs, allowing for all sorts of data to be analyzed that would otherwise need to be collected and reported manually. Otherwise difficult questions become easy to answer. Has this test, with this seed ever failed before? What is the host utilization like during a nightly regression? When did coverage drop off?

Figure 9: Test Output.


Figure 10: Regression Data Examples.


SUMMARY

Continuous Integration with Jenkins CI, coupled with Questa Verification Run Manager, provides a powerful automated solution for build and regression management. By automating the regression process and helping to identify problem areas earlier, they allow verification engineers to make more efficient use of the time given even in the tightest of schedules.

Back to Top

Table of Contents

Verification Horizons Articles:

  • A Good Plan, Plus Automation, Plus the Right Tools, Equals...Useful.

  • An Evaluation of the Advantages of Moving from a VHDL to a UVM Testbench

  • First Time Unit Testing Experience Report with SVUnit

  • The Verification Academy Patterns Library

  • Increased Efficiency with Questa® VRM and Jenkins Continuous Integration

  • Verifying Display Standards - A Comprehensive UVM-based Verification IP Solution

  • Nine Effective Features of NVMe® Questa® Verification IP to Help You Verify PCIe® Based SSD Storage

  • MIPI C-PHY™ - Man of the Hour

  • Total Recall: What to Look for in a Memory Model Library

  • Certus™ Silicon Debug: Don't Prototype Without It

  • Simplified UVM for FPGA Reliability UVM for "Sufficient Elemental Analysis" in DO-254 Flows

  • Complex Signal Processing Verification under DO-254 Constraints

  • Simplifying Generation of DO-254 Compliant Verification Documents for AEH Devices

  • DO-254 Compliant UVM VIP Development

  • Reusable Verification Framework

Related Resources

Jenkins Installation & Setup

Jenkins & Questa VRM Plug-in | Installation and Plug-in Setup | Darron May

In this session you will be introduced to the Jenkins continuous integration system, along with step by step installation and setup instructions.

Jenkins Project Configuration

Jenkins & Questa VRM Plug-in | Project Configuration | Darron May

In this session we will walk through the project configuration and how to setup a job with the Questa VRM plug-in.

Jenkins VRM Integration Demo

Jenkins & Questa VRM Plug-in | VRM Integration Demo | Darron May

In this session, we will demo the Questa VRM Jenkins integration and you will see first hand how to execute tests and generate reports.

Siemens Digital Industries Software

Siemens Digital Industries Software

##TodayMeetsTomorrow

Solutions

  • Cloud
  • Mendix
  • Siemens EDA
  • MindSphere
  • Siemens PLM
  • View all portfolio

Explore

  • Digital Journeys
  • Community
  • Blog
  • Online Store

Siemens

  • About Us
  • Careers
  • Events
  • News and Press
  • Newsletter
  • Customer Stories

Contact Us

USA:

phone-office +1 800 498 5351

See our Worldwide Directory

  • Contact Us
  • Support Center
  • Give us Feedback
©2021 Siemens Digital Industries Software. All Rights Reserved.
Terms of Use Privacy Cookie Policy