Increased Efficiency with Questa® VRM and Jenkins Continuous Integration
by Thomas Ellis, Mentor Graphics
"Time is really the only capital that any human being has, and the only thing he can't afford to lose."
—Thomas Edison
For all the incredible technological advances to date, no one has found a way to generate additional time. Consequently, there never seems to be enough of it. Since time cannot be created, it is utterly important to ensure that it is spent as wisely as possible. Applying automation to common tasks and identifying problems earlier are just two proven ways to best utilize time during the verification process. Continuous Integration (CI) is a software practice, which is focused on doing precisely that, resulting in a more efficient use of time.
WHAT IS CONTINUOUS INTEGRATION?
The basic principle behind Continuous Integration is that the longer a branch of code is checked out, the more it begins to drift away from what is stored in the repository. The more the two diverge, the more complicated it becomes to eventually merge in changes easily, ultimately leading to what is commonly referred to as "integration hell". To avoid this, and ultimately save engineers time, CI calls for integrating regularly and often (typically daily).
Figure 1: Continuous Integration Flow.
 |
Regular check-ins are of course, only half the equation, you need to be able to verify their changes quickly as well, otherwise many small check-ins over several days, is no different than one large check-in at weeks end. Commonly, in a Continuous Integration environment, a CI server monitors the source control for check-in's, which in turn triggers a CI process (time-based triggers are also common). This process will then build the necessary design files, and run the requisite integration tests. Once complete, the results of the tests are reported back to the user, and assuming everything passed, can now be safely committed to the repository.
By following this model, issues can be caught earlier in the development process, and can be resolved quicker as there is less variance between check-ins.
This practice has been used successfully for many years in the software industry, so much so, that it is fairly common place today. However, the idea of Continuous Integration is still fairly new in the realm of hardware verification, so it is difficult to find any published metrics on its usage as it pertains to that space specifically. However, one of the benefits of adopting a more mature technology, is you can avoid making some of the pitfalls which plagued early adopters. Since Continuous Integration technology has been used by software teams for some time, you can glean a general idea of both how widespread its usage has become, as well as what technologies have risen to the top.
ZeroTurnaround is a development company, which amongst other things, conducts an annual global survey of Java developers, and produces a report of the tools and technologies being most commonly used by the industry. In 2014, they received responses from nearly 2200 developers covering many topics, one of which was their usage of Continuous Integration Technologies. In that survey, they found that roughly 80% of (or four out of five) developers, reported using Continuous Integration in their teams. A number which itself, showed fairly significant growth, up from 68% the prior year.
Figure 2: 4 in 5 Developers Use Continuous Integration.
 |
Another interesting aspect of the report, is the breakdown of which Continuous Integration servers were most commonly used. Far and away the most popular server was Jenkins, which was reportedly used by 70% of the developers who claimed to use CI. The second place tool was used by a mere 9% of users. So what is Jenkins, and why is it the favorite CI tool of so many users?
MEET JENKINS
Jenkins is a freely available, open-source continuous integration tool (released under the MIT license).
Figure 3: Jenkins CI.
 |
A quick background, Jenkins was initially developed by Kohsuke Kawaguchi while he was working at Sun Microsystems in 2004. However, at the time, the project was named Hudson. After its initial release in 2005, it quickly became a favorite open-source build server. In 2010, issues began to arise between the open source community working on Hudson, and Oracle (who had since acquired Sun). Eventually requiring a vote to be called, as to whether or not to fork the project. Based on an overwhelmingly supportive community vote, Jenkins was created as a fork of Hudson. The majority of those working on, or using Hudson at the time, eventually migrated to Jenkins. Currently there are at least 127,000 installations of Jenkins (based on the anonymous usage statistics of the tool). Remember the ZeroTurnaround study? They found only 8% of users to still be using Hudson.?
Apart from being open-source, Jenkins is easy to install and highly configurable via its web interface. While Jenkins offers a lot itself, it is also highly extensible via plug-ins to the tool. At present, it boasts 1350+ plugins from 580+ contributors, to perform a myriad of different tasks, allowing for many third-party tools to leverage the power of Jenkins.
JENKINS AND VRM
On the surface, one might think that Jenkins and VRM are competitive technologies; after all, both tools can build, run and report on regressions. However, in actuality, they are truthfully complementary technologies. Furthermore, by marrying the two technologies together, you can benefit from the strengths of both, and create an extremely powerful solution for building and testing hardware designs.
While Jenkins is extremely flexible, and can run just about anything, with lots of neat bells and whistles to boot, nothing within the Jenkins core is knowledgeable about hardware verification. In the same way that VRM does not natively monitor code repositories for developer check-ins, concepts like merging SystemVerilog functional coverage, or recognizing why a UVM testbench failed are not native to Jenkins, in the way that they are at the core of VRM. What you want to do is leverage Jenkins' strengths as a build system to monitor our source repository and allow it to launch our regressions. Ultimately what it will launch though, is VRM, which will handle managing the individual verification tasks by integrating with our grid software, collecting and merging the coverage and results, etc. Once the regression is completed, Jenkins can then ask VRM to supply metrics for what was accomplished during the run, and display those results in its web dashboard.
Figure 4: Jenkins and VRM Work Together.
 |
A REGRESSION IN JENKINS
Let's take a quick look at setting up a project to run VRM in Jenkins. On the following page is the project configuration page in Jenkins.
Here you can see the basic steps for configuring a project in Jenkins. Tasks in Jenkins are represented by builds. A build could be a complete regression, it could be the running of unit tests, or any other task you may wish Jenkins to automate.
Figure 5: Configuring a Project in Jenkins.
 |
First you specify when to run our tests via a build trigger. A build trigger can be a period of time, a specific time, or you can even have Jenkins monitor your repository for changes, and automatically start a build for you. Jenkins will then run whatever you tell it to, which in this example, will be to launch VRM.
Finally, Jenkins will report the results of the regression run (or build). Out of the box, Jenkins will give you basic pass/fail information and some basic reporting of results, however, its lack of the metrics verification engineers are most commonly interested in, makes it feel a bit empty. To solve this shortcoming, you need one last piece to truly tie everything together neatly.
VRM JENKINS PLUG-IN
As mentioned earlier, one of the key benefits of Jenkins is that it is highly extensible through plug-ins. To get Jenkins to become more useful with respect to you running regressions with VRM, you can leverage the VRM Jenkins plug-in. You simply install the plug-in through Jenkins plug-in manager, and now Jenkins has the ability to understand code and functional coverage, determine where log files reside, monitor host utilization, and many other verification centric tasks.
To display the VRM results, and enable these features, you simply need to add what is called a post-build action (in Jenkins terms), which has Jenkins call the plug-in to make sense of the regression results.
The setup is very straightforward, you simply need to tell Jenkins where the regression ran. Additionally, you can optionally select to enable a few other features such as creating HTML reports and publishing a coverage graph to the project page. That's it! Jenkins and VRM will do the rest.
Figure 6: VRM Jenkins Plug-In.
 |
VRM REGRESSION RESULTS IN JENKINS
One of the great features of Jenkins is its web dashboard. Now that it is using the VRM Jenkins Plug-in, you get access to a lot of great information at a glance. There is far too much to show in this short article, but here are a few examples.
The main project page has two graphs which shows you a trend of the test results, as well as the coverage results from all your past regression runs. You also get a summary table which lists the last several regressions, including information on their duration, pass/fail statistics and coverage. There are also quick links to the HTML coverage report, as well as the latest test results.
If you dig into the most recent build, you can get more detailed data on that particular regression.
Figure 7: Jenkins Project Dashboard.
 |
Here, in addition to pass/fail and coverage results, you can also see a list of the specific tests which failed, providing a means for easy high-level inspection. Expanding a given test, will give us both the reason for the failure, as well as the standard output for the test in question.
Figure 8: Regression Results.
 |
The plug-in leverages the vast amount of data VRM collects from the regression runs, allowing for all sorts of data to be analyzed that would otherwise need to be collected and reported manually. Otherwise difficult questions become easy to answer. Has this test, with this seed ever failed before? What is the host utilization like during a nightly regression? When did coverage drop off?
Figure 9: Test Output.
 |
Figure 10: Regression Data Examples.
 |
SUMMARY
Continuous Integration with Jenkins CI, coupled with Questa Verification Run Manager, provides a powerful automated solution for build and regression management. By automating the regression process and helping to identify problem areas earlier, they allow verification engineers to make more efficient use of the time given even in the tightest of schedules.
Back to Top