INTRODUCTION
Perhaps my interest in data mining and analytics originated from Steven Levitt and Stephen Dubner's 2005 bestselling book Freakonomics: A Rogue Economist Explores the Hidden Side of Everything,. The authors in this book apply economic theory (a science of measurement) to a diverse set of subjects not usually covered by "traditional" economists; such as correlating cheating as applied to teachers and sumo wrestlers. This book inspired me to look at data differently. In that spirit, I decided to have some fun with the data from our 2016 Wilson Research Group Functional Verification Study[1] by examining interesting correlations in an attempt to uncover unexpected observations. For example, in the March 2015 issue of Verification Horizons, I decided to correlate design size with first silicon success from our previous industry study, and the results were non-intuitive. That is, the smaller the design the less the likelihood of achieving first silicon success.[2] This observation concerning design size and likelihood of achieving first silicon success still holds true today.
For this issue of Verification Horizons, I have decided to do a deeper dive into our 2016 industry study and see what other non-intuitive observations could be uncovered. Specifically, I wanted to answer the following questions: (1) Does verification maturity impact silicon success (in terms of functional quality)? (2) Does the adoption of safety critical design practices improve silicon success?
But before I answer these questions let's look at some general findings from our 2016 study.
RESOURCE TRENDS
It is assumed that the industry is experiencing growing resource demands due to rising design complexity. To validate this assumption let us examine a recent industry trend.
Figure 1 shows the percentage of total project time that is spent in functional verification. As you would expect, the results are all over the spectrum; whereas, some projects spend less time involved in functional verification, other projects spend more. The average total project time spent in functional verification in 2016 was 55 percent, which did not change significantly from 2014.
 |
Figure 1. Percentage of ASIC/IC Project Time Spent in Verification
Perhaps one of the biggest challenges in design and verification today is identifying solutions to increase productivity to control engineering head count. To illustrate the need for productivity improvement, we discuss the trend in terms of increasing engineering head count. Figure 2 below shows the mean peak number of engineers working on a project. Again, this is an industry average since some projects have many engineers while other projects have few. You can see that the mean peak number of verification engineers today is greater than the mean peak number of design engineers. In other words, there are, on average, more verification engineers working on a project than design engineers. This situation has changed significantly since 2007.
 |
Figure 2. Mean Peak Number of Design and Verification Engineers Working on a Project
Another way to comprehend the impact of today's project headcount trends is to calculate the compounded annual growth rate (CAGR) for both design and verification engineers. Between 2007 and 2016, the industry experienced a 3.6 percent CAGR for design engineers and a 10.4 percent CAGR for verification engineers. Clearly, the double-digit increase in required verification engineers has become a major project cost-management concern and is one indicator of growing verification effort.
But verification engineers are not the only project stakeholders involved in the verification process. Design engineers spend a significant amount of their time in verification too, as shown in Figure 3 below.
 |
Figure 3. Where Design Engineers Spend Their Time
In 2016, design engineers spent slightly more time in design activities, yet still a significant amount of time involved in verification. However, this is a reversal in the trends observed from the 2010 and 2012 studies, which indicated that design engineers spent more time in verification activities than design activities. The data suggest that design effort has risen since 2012 when you take into account that: (a) design engineers are spending more time in the design process, and (b) there was a nine percent CAGR in demand for design engineers between 2012 and 2014 (shown in Figure 3), which is a steeper increase than the overall 3.6 CAGR for design engineers spanning 2007 through 2016. So what is contributing to this increased design effort? Our 2016 study found an increase in the adoption and implementation of low-power features into the design, which might partially account for the increase in design effort.
Figure 4 shows where verification engineers spend their time (on average) for various tasks. We do not show trends here since this aspect of project resources was not studied prior to 2012, and there was not a statistically significant change in the results between 2012, 2014 and 2016.
 |
Figure 4. Where ASIC/IC Verification Engineers Spend Their Time
Our study found that verification engineers spend more of their time in debugging than any other activity. This is an important focus area of EDA R&D that is necessary for improving productivity and predictability within a project.
SCHEDULE AND RESPIN TRENDS
Today we find that a significant amount of effort is being applied to functional verification. An important question our study has tried to answer is whether this increasing effort is paying off. In this section, we present verification findings in terms of schedules and number of required spins.
Figure 5 presents the design completion time compared to the project's original schedule. The data originally suggested that 2014 saw a slight improvement in projects meeting their original schedule. However, our most recent study is consistent with the 2007 and 2012 findings. Regardless, meeting the originally planned schedule is still a challenge for most of the industry.
 |
Figure 5. Design Completion Compared to Original Schedule
Figure 6 shows the industry trend for the number of spins required between the start of a project and final production. Even though designs have increased in complexity, the data suggest that projects are not getting any worse in terms of the number of required spins before production. Still, only about 33 percent of today's projects are able to achieve first silicon success.
 |
Figure 6. Required Number of Spins
Although the focus of this article is on ASIC/IC designs, I think it is worth sharing an interesting finding related to FPGA designs, which is, the number of non-trivial bugs that escape into production and are found in the field. This is a useful metric to measure verification effectiveness for FPGA projects since there is no analogous metric for required ASIC/IC spins associated with FPGA designs. The results were surprising and are presented in Figure 7. Only 22 percent of today's FPGA design projects are able to produce designs without a non-trivial bug escaping into the final product. The reason this is significant is because for some market segments (such as safety critical designs) the cost of upgrading the FPGA in the field can be huge since this often requires a complete revalidation of the system.
 |
Figure 7. FPGA Non-Trivial Bug Escapes into Production
VERIFICATION MATURITY AND SILICON SUCCESS
Now let us examine data from our study and attempt to answer the first question, "Does verification maturity impact silicon success?" The approach we used to answer this question was to measure the adoption of various verification techniques related to ASIC/IC projects, and then we compare these results against achieving first silicon success. The data, in Figure 8, suggest that the more mature an ASIC/IC project is in its adoption of verification technology, the greater the likelihood of achieving first silicon success.
 |
Figure 8. ASIC/IC Spins and Verification Maturity
Similarly, in Figure 9 we examine the adoption of various verification techniques related to FPGA projects, and then we compare these results against non-trivial bugs that escape into production.
 |
Figure 9. FPGA Non-Trivial Bug Escapes into Production and Verification Maturity
Again, the results are statistically significant and suggest that the more mature an FPGA project is in its adoption of verification technology, the greater the likelihood that non-trivial bugs are prevented from escaping into production.
SAFETY CRITICAL DESIGNS AND SILICON SUCCESSES
Next we examine data from our study in an attempt to answer our second question, "Does the adoption of safety critical design practices improve silicon success?" Intuitively, one might think that a rigid and structured process required by the various safety critical development practices (such as DO-254 for mil/aero, ISO 26262 for automotive, IEC 60601, and so forth) would yield higher quality in terms of silicon success.
First, let us look at the percentage of ASIC/IC and FPGA projects that claimed to be working on a safety critical design, shown in Figure 10.
 |
Figure 10. Percentage of Projects Working on Safety Critical Designs
Next, Figure 11 compares the number of required spins for both safety critical and non-safety critical ASIC/IC designs, while Figure 12 compares the FPGA designs with non-trivial bug escapes for both safety critical and non-safety critical designs.
 |
Figure 11. Required ASIC/IC Spins for Safety Critical vs. Non-Safety Critical Designs
 |
Figure 12. Non-Trivial Bug Escapes for Safety Critical vs. Non-Safety Critical FPGA Designs
You can see that 72 percent of safety critical ASIC/IC designs require a respin, while 78 percent of safety critical FPGA designs had bugs escape into production. Clearly, the process developed to ensure safety does not necessarily ensure quality.
To be fair, however, many of the safety features implemented in designs are quite complex and increase the verification burden.
CONCLUSION
In this article, I presented a few highlights from the 2016 Wilson Research Group Functional Verification Study that was commissioned by Mentor Graphics. One of the key takeaways from our study is that verification effort continues to increase, which was observed by the double-digit CAGR increase in peak number verification engineers required on a project.
In general, the industry is maturing its verification processes as witnessed by the verification technology adoption trends. However, just like our 2014 observation, we found that in 2016, smaller designs were less likely to achieve first silicon success.
For this article I decided to do a deeper dive into analyzing the data from our new study in an attempt to answer the following questions: (1) Does verification maturity impact silicon success? (2) Does the adoption of safety critical design practices improve silicon success? For our first question we found that the data suggest that the more mature a project is in adopting advanced functional verification techniques, the greater the likelihood of silicon success (in terms of functional quality). For our second question we found that the adoption of one of the industry's safety critical development practices (such as, DO-254, ISO 26262, IEC 60601, and so forth) does not necessarily improve silicon success.
As a closing comment, all the findings from the 2016 Wilson Research Group Functional Verification Study have been released through a series of Verification Horizons blogs[1], as well as an article I recently published at DVCon 2017.[3]
REFERENCES
- H. Foster, (2016, August, 8), Prologue: The 2016 Wilson Research Group Functional Verification Study
- H. Foster, Verification Horizons, Does Design Size Influence First Silicon Success?, March 2015, Volume 11, Issue 1
- H. Foster, Trends in Functional Verification: A 2016 Industry Study. In the Proceedings of the 2017 DVCon
Back to Top