Verification Academy

Search form

My Account Menu

  • Register
  • Log In
  • Topics
  • Courses
  • Forums
  • Patterns Library
  • Cookbooks
  • Events
  • More
  • All Topics
    The Verification Academy offers users multiple entry points to find the information they need. One of these entry points is through Topic collections. These topics are industry standards that all design and verification engineers should recognize. While we continue to add new topics, users are encourage to further refine collection information to meet their specific interests.
    • Languages & Standards

      • Portable Test and Stimulus
      • Functional Safety
      • Design & Verification Languages
    • Methodologies

      • UVM - Universal Verification Methodology
      • UVM Framework
      • UVM Connect
      • FPGA Verification
      • Coverage
    • Techniques & Tools

      • Verification IP
      • Simulation-Based Techniques
      • Planning, Measurement, and Analysis
      • Formal-Based Techniques
      • Debug
      • Clock-Domain Crossing
      • Acceleration
  • All Courses
    The Verification Academy is organized into a collection of free online courses, focusing on various key aspects of advanced functional verification. Each course consists of multiple sessions—allowing the participant to pick and choose specific topics of interest, as well as revisit any specific topics for future reference. After completing a specific course, the participant should be armed with enough knowledge to then understand the necessary steps required for maturing their own organization’s skills and infrastructure on the specific topic of interest. The Verification Academy will provide you with a unique opportunity to develop an understanding of how to mature your organization’s processes so that you can then reap the benefits that advanced functional verification offers.
    • Universal Verification Methodology (UVM)

      • Advanced UVM
      • Basic UVM
      • Introduction to UVM
      • UVM Connect
      • UVM Debug
      • UVMF - One Bite at a Time
    • Featured Courses

      • Introduction to ISO 26262
      • Introduction to DO-254
      • Clock-Domain Crossing Verification
      • Portable Stimulus Basics
      • Power Aware CDC Verification
      • Power Aware Verification
      • SystemVerilog OOP for UVM Verification
    • Additional Courses

      • Assertion-Based Verification
      • An Introduction to Unit Testing with SVUnit
      • Evolving FPGA Verification Capabilities
      • Metrics in SoC Verification
      • SystemVerilog Testbench Acceleration
      • Testbench Co-Emulation: SystemC & TLM-2.0
      • Verification Planning and Management
      • VHDL-2008 Why It Matters
    • Formal-Based Techniques

      • Formal Assertion-Based Verification
      • Formal-Based Technology: Automatic Formal Solutions
      • Formal Coverage
      • Getting Started with Formal-Based Technology
      • Handling Inconclusive Assertions in Formal Verification
      • Sequential Logic Equivalence Checking
    • Analog/Mixed Signal

      • AMS Design Configuration Schemes
      • Improve AMS Verification Performance
      • Improve AMS Verification Quality
  • All Forum Topics
    The Verification Community is eager to answer your UVM, SystemVerilog and Coverage related questions. We encourage you to take an active role in the Forums by answering and commenting to any questions that you are able to.
    • UVM Forum

      • Active Questions
      • Solutions
      • Replies
      • No Replies
      • Search
      • UVM Forum
    • SystemVerilog Forum

      • Active Questions
      • Solutions
      • Replies
      • No Replies
      • Search
      • SystemVerilog Forum
    • Coverage Forum

      • Active Questions
      • Solutions
      • Replies
      • No Replies
      • Search
      • Coverage Forum
    • Additional Forums

      • Announcements
      • Downloads
      • OVM Forum
  • Patterns Library
    The Verification Academy Patterns Library contains a collection of solutions to many of today's verification problems. The patterns contained in the library span across the entire domain of verification (i.e., from specification to methodology to implementation—and across multiple verification engines such as formal, simulation, and emulation).
    • Implementation Patterns

      • Environment Patterns
      • Stimulus Patterns
      • Analysis Patterns
      • All Implementation Patterns
    • Specification Patterns

      • Occurrence Property Patterns
      • Order Property Patterns
      • All Specification Patterns
    • Pattern Resources

      • Start Here - Patterns Library Overview
      • Whitepaper - Taking Reuse to the Next Level
      • Verification Horizons - The Verification Academy Patterns Library
      • Contribute a Pattern to the Library
  • All Cookbooks
    Find all the methodology you need in this comprehensive and vast collection. The UVM and Coverage Cookbooks contain dozens of informative, executable articles covering all aspects of UVM and Coverage.
    • UVM Cookbook

      • UVM Basics
      • Testbench Architecture
      • DUT-Testbench Connections
      • Configuring a Test Environment
      • Analysis Components & Techniques
      • End Of Test Mechanisms
      • Sequences
      • The UVM Messaging System
      • Other Stimulus Techniques
      • Register Abstraction Layer
      • Testbench Acceleration through Co-Emulation
      • Debug of SV and UVM
      • UVM Connect - SV-SystemC interoperability
      • UVM Versions and Compatibility
      • UVM Cookbook
    • Coding Guidelines & Deployment

      • Code Examples
      • UVM Verification Component
      • Package/Organization
      • Questa/Compiling UVM
      • SystemVerilog Guidelines
      • SystemVerilog Performance Guidelines
      • UVM Guidelines
      • UVM Performance Guidelines
    • Coverage Cookbook

      • Introduction
      • What is Coverage?
      • Kinds of Coverage
      • Specification to Testplan
      • Testplan to Functional Coverage
      • Bus Protocol Coverage
      • Block Level Coverage
      • Datapath Coverage
      • SoC Coverage Example
      • Requirements Writing Guidelines
      • Coverage Cookbook
  • All Events
    No one argues that the challenges of verification are growing exponentially. What is needed to meet these challenges are tools, methodologies and processes that can help you transform your verification environment. These recorded seminars from Verification Academy trainers and users provide examples for adoption of new technologies and how to evolve your verification process.
    • Upcoming & Featured Events

      • Low Power Verification - 4/29
      • Fault Campaign for Mixed-Signal - 5/4
      • User2User - 5/26
      • Webinar Calendar
    • On-Demand Webinars

      • CDC+RDC Analysis
      • Basic Abstraction Techniques
      • Safety Analysis Techniques
      • QVIP Workflow and Debug for PCIe
      • Writing a Proxy-driven Testbench
      • Achieving High Defect Coverage
      • Visualizer Features
      • All On-Demand Webinars
    • Recording Archive

      • Siemens EDA 2021 Functional Verification Webinar Series
      • Improving Your SystemVerilog & UVM Skills
      • Should I Kill My Formal Run?
      • Visualizer Debug Environment
      • Industry Data & Surveys
      • All Recordings
    • Conferences

      • DVCon 2021
      • DVCon 2020
      • DAC 2019
      • All Conferences
    • Mentor Learning Center

      • SystemVerilog Fundamentals
      • SystemVerilog UVM
      • View all Learning Paths
  • About Verification Academy
    The Verification Academy will provide you with a unique opportunity to develop an understanding of how to mature your organization's processes so that you can then reap the benefits that advanced functional verification offers.
    • Blog & News

      • Verification Horizons Blog
      • Academy News
      • Academy Newsletter
      • Technical Resources
    • Verification Horizons Publication

      • Verification Horizons - March 2021
      • Verification Horizons - November 2020
      • Verification Horizons - July 2020
      • Issue Archive
    • About Us

      • Verification Academy Overview
      • Subject Matter Experts
      • Contact Us
    • Training

      • Questa Basic
      • Questa Advanced
      • Mastering Questa
  • Home
  • Verification Horizons
  • March 2017
  • Will Safety Critical Design Practices Improve First Silicon Success?

Will Safety Critical Design Practices Improve First Silicon Success?

Verification Horizons - Tom Fitzpatrick, Editor

Will Safety Critical Design Practices Improve First Silicon Success? by Harry D. Foster, Mentor Graphics

INTRODUCTION

Perhaps my interest in data mining and analytics originated from Steven Levitt and Stephen Dubner's 2005 bestselling book Freakonomics: A Rogue Economist Explores the Hidden Side of Everything,. The authors in this book apply economic theory (a science of measurement) to a diverse set of subjects not usually covered by "traditional" economists; such as correlating cheating as applied to teachers and sumo wrestlers. This book inspired me to look at data differently. In that spirit, I decided to have some fun with the data from our 2016 Wilson Research Group Functional Verification Study[1] by examining interesting correlations in an attempt to uncover unexpected observations. For example, in the March 2015 issue of Verification Horizons, I decided to correlate design size with first silicon success from our previous industry study, and the results were non-intuitive. That is, the smaller the design the less the likelihood of achieving first silicon success.[2] This observation concerning design size and likelihood of achieving first silicon success still holds true today.

For this issue of Verification Horizons, I have decided to do a deeper dive into our 2016 industry study and see what other non-intuitive observations could be uncovered. Specifically, I wanted to answer the following questions: (1) Does verification maturity impact silicon success (in terms of functional quality)? (2) Does the adoption of safety critical design practices improve silicon success?

But before I answer these questions let's look at some general findings from our 2016 study.

RESOURCE TRENDS

It is assumed that the industry is experiencing growing resource demands due to rising design complexity. To validate this assumption let us examine a recent industry trend.

Figure 1 shows the percentage of total project time that is spent in functional verification. As you would expect, the results are all over the spectrum; whereas, some projects spend less time involved in functional verification, other projects spend more. The average total project time spent in functional verification in 2016 was 55 percent, which did not change significantly from 2014.

Figure 1. Percentage of ASIC/IC Project Time Spent in Verification


Perhaps one of the biggest challenges in design and verification today is identifying solutions to increase productivity to control engineering head count. To illustrate the need for productivity improvement, we discuss the trend in terms of increasing engineering head count. Figure 2 below shows the mean peak number of engineers working on a project. Again, this is an industry average since some projects have many engineers while other projects have few. You can see that the mean peak number of verification engineers today is greater than the mean peak number of design engineers. In other words, there are, on average, more verification engineers working on a project than design engineers. This situation has changed significantly since 2007.

Figure 2. Mean Peak Number of Design and Verification Engineers Working on a Project


Another way to comprehend the impact of today's project headcount trends is to calculate the compounded annual growth rate (CAGR) for both design and verification engineers. Between 2007 and 2016, the industry experienced a 3.6 percent CAGR for design engineers and a 10.4 percent CAGR for verification engineers. Clearly, the double-digit increase in required verification engineers has become a major project cost-management concern and is one indicator of growing verification effort.

But verification engineers are not the only project stakeholders involved in the verification process. Design engineers spend a significant amount of their time in verification too, as shown in Figure 3 below.

Figure 3. Where Design Engineers Spend Their Time


In 2016, design engineers spent slightly more time in design activities, yet still a significant amount of time involved in verification. However, this is a reversal in the trends observed from the 2010 and 2012 studies, which indicated that design engineers spent more time in verification activities than design activities. The data suggest that design effort has risen since 2012 when you take into account that: (a) design engineers are spending more time in the design process, and (b) there was a nine percent CAGR in demand for design engineers between 2012 and 2014 (shown in Figure 3), which is a steeper increase than the overall 3.6 CAGR for design engineers spanning 2007 through 2016. So what is contributing to this increased design effort? Our 2016 study found an increase in the adoption and implementation of low-power features into the design, which might partially account for the increase in design effort.

Figure 4 shows where verification engineers spend their time (on average) for various tasks. We do not show trends here since this aspect of project resources was not studied prior to 2012, and there was not a statistically significant change in the results between 2012, 2014 and 2016.

Figure 4. Where ASIC/IC Verification Engineers Spend Their Time


Our study found that verification engineers spend more of their time in debugging than any other activity. This is an important focus area of EDA R&D that is necessary for improving productivity and predictability within a project.

SCHEDULE AND RESPIN TRENDS

Today we find that a significant amount of effort is being applied to functional verification. An important question our study has tried to answer is whether this increasing effort is paying off. In this section, we present verification findings in terms of schedules and number of required spins.

Figure 5 presents the design completion time compared to the project's original schedule. The data originally suggested that 2014 saw a slight improvement in projects meeting their original schedule. However, our most recent study is consistent with the 2007 and 2012 findings. Regardless, meeting the originally planned schedule is still a challenge for most of the industry.

Figure 5. Design Completion Compared to Original Schedule


Figure 6 shows the industry trend for the number of spins required between the start of a project and final production. Even though designs have increased in complexity, the data suggest that projects are not getting any worse in terms of the number of required spins before production. Still, only about 33 percent of today's projects are able to achieve first silicon success.

Figure 6. Required Number of Spins


Although the focus of this article is on ASIC/IC designs, I think it is worth sharing an interesting finding related to FPGA designs, which is, the number of non-trivial bugs that escape into production and are found in the field. This is a useful metric to measure verification effectiveness for FPGA projects since there is no analogous metric for required ASIC/IC spins associated with FPGA designs. The results were surprising and are presented in Figure 7. Only 22 percent of today's FPGA design projects are able to produce designs without a non-trivial bug escaping into the final product. The reason this is significant is because for some market segments (such as safety critical designs) the cost of upgrading the FPGA in the field can be huge since this often requires a complete revalidation of the system.

Figure 7. FPGA Non-Trivial Bug Escapes into Production


VERIFICATION MATURITY AND SILICON SUCCESS

Now let us examine data from our study and attempt to answer the first question, "Does verification maturity impact silicon success?" The approach we used to answer this question was to measure the adoption of various verification techniques related to ASIC/IC projects, and then we compare these results against achieving first silicon success. The data, in Figure 8, suggest that the more mature an ASIC/IC project is in its adoption of verification technology, the greater the likelihood of achieving first silicon success.

Figure 8. ASIC/IC Spins and Verification Maturity


Similarly, in Figure 9 we examine the adoption of various verification techniques related to FPGA projects, and then we compare these results against non-trivial bugs that escape into production.

Figure 9. FPGA Non-Trivial Bug Escapes into Production and Verification Maturity


Again, the results are statistically significant and suggest that the more mature an FPGA project is in its adoption of verification technology, the greater the likelihood that non-trivial bugs are prevented from escaping into production.

SAFETY CRITICAL DESIGNS AND SILICON SUCCESSES

Next we examine data from our study in an attempt to answer our second question, "Does the adoption of safety critical design practices improve silicon success?" Intuitively, one might think that a rigid and structured process required by the various safety critical development practices (such as DO-254 for mil/aero, ISO 26262 for automotive, IEC 60601, and so forth) would yield higher quality in terms of silicon success.

First, let us look at the percentage of ASIC/IC and FPGA projects that claimed to be working on a safety critical design, shown in Figure 10.

Figure 10. Percentage of Projects Working on Safety Critical Designs


Next, Figure 11 compares the number of required spins for both safety critical and non-safety critical ASIC/IC designs, while Figure 12 compares the FPGA designs with non-trivial bug escapes for both safety critical and non-safety critical designs.

Figure 11. Required ASIC/IC Spins for Safety Critical vs. Non-Safety Critical Designs


Figure 12. Non-Trivial Bug Escapes for Safety Critical vs. Non-Safety Critical FPGA Designs


You can see that 72 percent of safety critical ASIC/IC designs require a respin, while 78 percent of safety critical FPGA designs had bugs escape into production. Clearly, the process developed to ensure safety does not necessarily ensure quality.

To be fair, however, many of the safety features implemented in designs are quite complex and increase the verification burden.

CONCLUSION

In this article, I presented a few highlights from the 2016 Wilson Research Group Functional Verification Study that was commissioned by Mentor Graphics. One of the key takeaways from our study is that verification effort continues to increase, which was observed by the double-digit CAGR increase in peak number verification engineers required on a project.

In general, the industry is maturing its verification processes as witnessed by the verification technology adoption trends. However, just like our 2014 observation, we found that in 2016, smaller designs were less likely to achieve first silicon success.

For this article I decided to do a deeper dive into analyzing the data from our new study in an attempt to answer the following questions: (1) Does verification maturity impact silicon success? (2) Does the adoption of safety critical design practices improve silicon success? For our first question we found that the data suggest that the more mature a project is in adopting advanced functional verification techniques, the greater the likelihood of silicon success (in terms of functional quality). For our second question we found that the adoption of one of the industry's safety critical development practices (such as, DO-254, ISO 26262, IEC 60601, and so forth) does not necessarily improve silicon success.

As a closing comment, all the findings from the 2016 Wilson Research Group Functional Verification Study have been released through a series of Verification Horizons blogs[1], as well as an article I recently published at DVCon 2017.[3]

REFERENCES

  1. H. Foster, (2016, August, 8), Prologue: The 2016 Wilson Research Group Functional Verification Study
  2. H. Foster, Verification Horizons, Does Design Size Influence First Silicon Success?, March 2015, Volume 11, Issue 1
  3. H. Foster, Trends in Functional Verification: A 2016 Industry Study. In the Proceedings of the 2017 DVCon

Back to Top

Table of Contents

Verification Horizons Articles:

  • Historic Wins Begin with Preparing for Success.

  • Will Safety Critical Design Practices Improve First Silicon Success?

  • A Practical Methodology for Meeting ISO 26262 Random Faults Safety Goals in Automotive Semiconductor Products

  • Bridging UVM to the Portable Stimulus Standard with Questa® inFact

  • Automating Tests with Portable Stimulus from IP to SoC Level

  • UVM Tips and Tricks

  • Artifacts of Custom Checkers in Questa® Power Aware Dynamic Simulation

  • Complementing Functional Verification Through the Use of Available Timing Information

Siemens Digital Industries Software

Siemens Digital Industries Software

##TodayMeetsTomorrow

Solutions

  • Cloud
  • Mendix
  • Siemens EDA
  • MindSphere
  • Siemens PLM
  • View all portfolio

Explore

  • Digital Journeys
  • Community
  • Blog
  • Online Store

Siemens

  • About Us
  • Careers
  • Events
  • News and Press
  • Newsletter
  • Customer Stories

Contact Us

USA:

phone-office +1 800 547 3000

See our Worldwide Directory

  • Contact Us
  • Support Center
  • Give us Feedback
©2021 Siemens Digital Industries Software. All Rights Reserved.
Terms of Use Privacy Cookie Policy