So You Think You Have Good Stimulus: System-Level Distributed Metrics Analysis and Results
-
Body
Authors: Alan Hunter - ARM®Andreas Meyer - Mentor Graphics Abstract: Developing and maintaining an effective and efficient verification suite for a complex system requires the ability to measure, understand, and improve the environment. Distributed, hierarchical caches are an example of interacting components within an SoC. Understanding how well the components are verified is a challenge since the cache interactions are complex, the components are distributed across an environment, and the data is spread across one or more regressions. This paper discusses the challenges of collecting metrics, providing the visualization to understand complex state machine interactions, and then reviews results of a regression analysis. A complex ARM-based SoC with multiple processors, and a coherent distributed multi-level cache is the basis of our study. A modern constrained random test suite is used to generate regression suites, with traditional code and functional coverage methods used to steer and grade the test suite. While this approach has been successful, it does require significant compute resources to reach coverage closure, and it has been difficult to determine how to improve the efficiency and quality of the test suite. We introduce statistical coverage as an approach to provide new coverage analysis capability. Within our ARM SoC project, we show how we are able to find and fix significant weaknesses in the stimulus that could not be seen using traditional code and functional coverage metrics. The stimulus improvements provided improved regression efficiency, found areas that had not been fully explored, and as a result found additional RTL issues. Introduction: Developing effective stimulus in an SoC environment, with complex interactions between large IP blocks can be challenging. Current methods for verification are not keeping up with complexity trends in modern SoCs such as ARM processor environments. A common verification approach of constrained random stimulus generation relies on coverage metrics to determine completeness, and provide the feedback needed to modify the constraints for full coverage. Current coverage methods are well suited for pointing out areas of code that haven't executed, or how well all branches of a state machine have been exercised. While existing coverage tools can even work reasonably well at measuring across a few pre-defined areas, there are currently no standard methods to capture interaction coverage of state machines distributed across multiple IP blocks. Metrics have been used for some time to gather more detailed information on many aspects of verification performance and effectiveness. We have applied the approach outlined earlier [1] to generate a new view into the performance of an SoC stimulus suite. Using real-world SoC development projects implementing state-of-the-art constrained random stimulus, we compared conventional coverage methods with new statistical coverage methods. We looked at example reports from the statistical coverage tool, the types of system-level information that it can report, and what the statistical coverage told us about the existing constrained-random test suite. The new measurement capability gave us information on where the stimulus constraints needed improvement, in addition to uncovering some surprisingly trivial bugs. With new coverage information, it was possible to modify the stimulus constraints and rerun the test suite. We'll show the results from running the improved stimulus, including examples of new functional errors that were uncovered, and statistical coverage reports showing the coverage improvements and a corresponding higher density of system-level interactions within the system. View & Download: Read the entire So You Think You Have Good Stimulus: System-Level Distributed Metrics Analysis and Results technical paper. Source: DVCon 2014