The 2019 global semiconductor market was valued at $385.4 billion after experiencing a 15% decline due to a 32% drop in the memory IC market, which is expected to recover in 2021. The FPGA portion of the semiconductor market is valued at about $5 billion. The FPGA semiconductor market is expected to reach a value of $7.5 billion by 2030, growing at a compounded annual growth rate (CAGR) of 4.4% during this forecast period. The growth in this market is being driven by new and expanding end-user applications related to data center computing, networking, and storage, as well as communication.
Historically, FPGAs have offered two primary advantages over ASICs. First, due to their low NRE, FPGAs are generally more cost effective than IC/ASICs for low-volume production. Second, FPGAs’ rapid prototyping capabilities and flexibility can reduce the development schedule since a majority of the verification and validation cycles have traditionally been performed in the lab. More recently, FPGAs offer advantages related to performance for certain accelerated applications by exploiting hardware parallelism (e.g., AI Neural Networks).
The IC/ASIC market in the mid- to late-2000 time frame underwent growing pains to address increased verification complexity. We find similar trends occurring in today’s FPGA market do to growing complexity. With the increased capacity and capability of today’s complex FPGAs and the emergence of high-performance SoC programmable FPGAs (e.g., Xilinx Zynq® UltraSCALE+, Intel® Stratix®, and Microsemi SmartFusion® 2), traditional lab-based approaches to FPGA verification and validation are becoming less effective. In this article, I quantify the ineffectiveness of today’s FPGA verification processes in terms of non-trivial bug escapes into production.
FPGA VERIFICATION EFFECTIVENESS
In this section, we present various FPGA project results in terms of verification effectiveness.
Non-Trivial Bug Escapes
IC/ASIC projects have often used the metric “number of required spins before production” as a benchmark to assess a project’s verification effectiveness. Historically, about 30% of IC/ASIC projects are able to achieve first silicon success, and most successful designs are productized on the second silicon spin. Unfortunately, FPGA projects have no equivalent metric. As an alternative to IC/ASIC spins, our study asked the FPGA participants “how many non-trivial bugs escaped into production?” The results shown in figure 1 are somewhat disturbing. In 2020, only 17% of all FPGA projects were able to achieve no bug escapes into production, which is worse than IC/ASIC in terms of first silicon success, and for some market segments, the cost of field repair can be significant. For example, in the mil-aero market, once a cover has been removed on a system to upgrade the FPGA, the entire system needs to be revalidated.
Figure 1 - Non-trivial FPGA bug escapes into production
Types of Flaws Resulting in Non-Trivial Bug Escapes
Figure 2 shows various categories of design flaws contributing to FPGA non-trivial bug escapes. The percentage of “logic or functional flaws” remains the leading cause of bugs. New flaws being tracked in the 2020 study are associated with safety (8%) and security (6%) features. Obviously multiple flaws can contribute to bug escapes, which is the reason the total percentage of flaws sums to more than 100%.
Figure 2 - Types of flaws resulting in FPGA bug escapes
Design Completion Compared to Original Schedule
In addition to bug escape metrics that we used to determine an FPGA project’s effectiveness, another metric we tracked was project completion compared to the original schedule, as shown in figure 3. Here we found that 68% of FPGA projects were behind schedule. One indication of growing design and verification complexity is reflected in the increasing number of FPGA projects missing schedule by more than 50% during the period 2014 through 2020.
Figure 3 - Actual FPGA project completion compared to original schedule
FPGA VERIFICATION EFFORT
In this section, we discuss trends in terms of FPGA project time and resources.
Percentage of Project Time Spent in Verification
Figure 4 shows the percentage of total FPGA project time spent in verification. You can see two extremes in this graph. In general, projects that spend very little time in verification are typically working on designs with a good deal of existing pre-verified design IP, which is integrated to create a new product. On the other extreme, projects that spend a significant amount of time in verification often have a high percentage of newly developed design IP that must be verified.
Overall, we found an increase in the average percentage of FPGA project time spent in verification during the period 2014 through 2020. Again, this is an indication of growing design and verification complexity.
Figure 4 - Percentage of FPGA project time spent in verification
Mean Peak Number of Engineers
Perhaps one of the biggest challenges today is to control cost and engineering head count, which means identifying FPGA design and verification solutions that increase productivity. To illustrate the need for productivity improvement, we discuss the trend in terms of increasing engineering head count. Figure 5 shows the mean peak number of FPGA engineers working on a project.
Figure 5 - Mean peak number of FPGA engineers
While, on average, the demand for FPGA design engineers grew at about a 1.5% CAGR between 2012 and 2020, the demand for FPGA verification engineers grew at a 5.5% CAGR. It is worth noting that during the period 2007 through 2014, the IC/ASIC market went through similar growth demands related to verification engineers to address growing verification complexity. While, on average, the demand for FPGA design engineers grew at about a 1.5% CAGR between 2012 and 2020, the demand for FPGA verification engineers grew at a 5.5% CAGR. It is worth noting that during the period 2007 through 2014, the IC/ASIC market went through similar growth demands related to verification engineers to address growing verification complexity.
But verification engineers are not the only project stakeholders involved in the verification process. Design engineers spend a significant amount of their time in verification too, as shown in figure 6. In 2020, design engineers spent on average 53% of their time involved in design activities and 47% of their time in verification. However, when compared to 2014 and 2016, the data indicate a trend showing that FPGA design engineers are now spending slightly less time involved in verification tasks. There are two reasons for this trend. First, many FPGA projects have added verification engineers to their teams, which means design engineers can focus most of their effort on design. Second, in general, there has been increased adoption of larger, more complex FPGAs, which has increased the design engineer’s workload.
Figure 6 - Where FPGA design engineers spend their time
Figure 7 shows where verification engineers spend their time (on average). Our study found that FPGA verification engineers spend more of their time debugging than with any other activity. From a management perspective, this can be a significant challenge when planning future projects’ effort and schedule based on previous projects’ data since debugging is unpredictable and varies significantly between projects.
Figure 7 - Where FPGA verification engineers spend their time
FPGA VERIFICATION ADOPTION TRENDS
To address growing verification complexity, we find that many FPGA projects are starting to mature their pre-lab functional verification processes. In this section, we present FPGA trends related to the adoption of various verification techniques, which are fairly standard practice today on most IC/ASIC projects.
Languages and Methodology Adoption Trends
In figure 8, we show the adoption trends for languages to build testbenches.
Figure 8 - WFPGA project verification language adoption
It is not uncommon for FPGA projects to use multiple languages when constructing their testbenches, which is why the percentage adoption sums to more than 100%. This practice of adopting multiple languages is often due to legacy code as well as purchased verification IP written in a different language. It is not uncommon for FPGA projects to use multiple languages when constructing their testbenches, which is why the percentage adoption sums to more than 100%. This practice of adopting multiple languages is often due to legacy code as well as purchased verification IP written in a different language.
Historically, VHDL was the predominant language used for FPGA testbench development, but we have recently seen increasing interest in SystemVerilog adoption. Today, it is not unusual to find that the RTL design was created using VHDL, and the testbench was created using SystemVerilog.
What is unusual in the 2020 data was the huge increase in C/C++ for testbench development compared to previous years. It is unknown at this point if this was an anomaly in this year’s study or an emerging trend. In addition, the 2018 level of adoption for the Accellera Portable Test and Stimulus Standard (PSS) was likely a misunderstanding by the study participants since its standardization was occurring at the same time as the 2018 study and few vendors supported it at that point in time. Finally, in 2020, for the first time, we explicitly asked about the adoption of Python for testbench development. In previous studies, Python was included with OTHER, which we now see has declined after moving Python to its own option.
The adoption trends for various base-class library and methodology standards are shown in figure 9, and we found that the Accellera UVM is currently the predominant standard that has been adopted to create FPGA testbenches. In 2018, we first started tracking the Open Source VHDL Verification Methodology™ (OSVVM) and the Universal VHDL Verification Methodology (UVVM), and in 2020 we are showing trends for the first time. In addition, for the 2020 study, we track Python-based methodologies, such as cocotb, for the first time.
Figure 9 - FPGA project methodology and base-class libraries adoption
Finally, FPGA project adoption trends for various assertion language standards are shown in figure 10. Finally, FPGA project adoption trends for various assertion language standards are shown in figure 10.
Figure 10 - FPGA project assertion language adoption
SystemVerilog Assertions (SVA) is the predominant assertion language in use today. Similar to languages used to build testbenches, it is not unusual to find FPGA projects create their RTL in VHDL and then create their assertions using SVA.
Technology Adoption Trend
The adoption trends for formal property checking (e.g., model checking) and automatic formal applications are shown in figure 11.
Figure 11 - FPGA project formal technology adoption trends
We found that the adoption of formal property checking on FPGA projects is growing at an impressive 10% CAGR, and the adoption of automatic formal applications is growing at a 21% CAGR. Historically, the formal property checking process has required specialized skills and expertise. However, the recent emergence of automatic formal applications provides narrowly focused solutions and does not require specialized skills for adoption. In general, formal solutions (i.e., formal property checking combined with automatic formal applications) is one of the fastest growing segments in functional verification in terms of project adoption.
Figure 12 shows the FPGA project adoption trends for various simulation-based techniques from 2012 through 2018, which include code coverage, functional coverage, assertions, and constrained-random simulation.
Figure 12 - FPGA project simulation technique trends
One observation from these adoption trends is that the FPGA market is starting to mature its verification processes. This maturity is likely due to the growing complexity of designs as discussed in the previous section.
CONCLUSION AND DISCUSSION
In this report, we presented FPGA design and verification trends based on a recent, large industry study. FPGAs have recently grown in complexity equal to many of today’s IC/ASIC designs. We quantified the impact of this growing complexity in terms of verification effectiveness and effort.
Perhaps the most disturbing finding from this year’s study relates to the number of FPGA projects with non-trivial bug escapes into production. We did find an interesting correlation between the improvement of reduced functional flaws contributing to non-trivial bug escapes, as shown in figure 1, and the maturing of FPGA projects’ functional verification processes, as previously discussed.
The data suggest that projects that are more mature in their functional verification processes will likely experience fewer bug escapes. To test this claim, we partitioned the study participants into two independent groups: FPGA projects with no bug escapes and FPGA projects that experienced a bug escape. We then examined the percentage adoption of various verification techniques and the results are shown in figure 13. What we are unable to measure from our study is how effective a project was in adopting any of these processes. Nonetheless, these findings are statistically significant in that the group with no bug escapes tended to have higher adoption of various verification techniques, which suggests they are more mature in their verification process.
Figure 13 - FPGA simulation technique adoption vs non-trival bug escapes
- IC-Insights, April Update to The McCLEAN REPORT 2020 EDITION.
- International Business Strategies, Semiconductor Market Analysis, Volume 29, No. 1, January, 2020.
- S. Trimberger, Three ages of FPGAs: a retrospective on the first thirty years of FPGA Technology, Proceedings of the IEEE, Vol 103, Issue 3, March 2015.
- H. Foster, "Why the Design Productivity Gap Never Happened", Proceedings of the International Conference on Computer-Aided Design (ICCAD), IEEE Press. pp. 581-58, 2013.
Back to Top