Tackling Random Blind Spots with Strategy-Driven Stimulus Generation
-
Body
Author: Matthew Ballance - Mentor Graphics Abstract: Hardware verification typically uses two primary types of stimulus generation: engineer-directed stimulus generation and open-loop random generation. This paper proposes a third approach for generating stimulus that automatically identifies and produces high-value test patterns from a constraint-based stimulus model. A trial implementation built on top of an intelligent testbench automation tool is described. Introduction: In today's hardware-verification space, two categories of tests are being created: engineer-directed tests and open-loop random tests. Directed and coverage-driven testing are examples of engineer-directed activities. Based on a test plan, the engineer lays out a goal of cases to test and either proceeds to create a test to exercise those cases (directed test) or defines functional coverage to ensure that random stimulus hits the identified test goal. Engineer-directed testing has many benefits, including ensuring that key functionality is verified in a documented and repeatable manner. The primary drawback is the "imagination gap": the very human inability to imagine key but obscure combinations of functionality that must be verified. In contrast, the main benefit of open-loop random simulation is that it closes this gap. Taking the engineer out the imagination loop enables automation to create legal but obscure cases and helps to find bugs. Both engineer-directed and open-loop random testing are valuable techniques. Consequently, a typical verification cycle starts with engineer-directed tests to verify basic functionality, transitions to a 'random simulations' stage to help find bugs, and concludes with closing coverage on the engineer-defined functional coverage goals. There are, of course, challenges in each phase of this cycle. This paper focuses on the random regressions phase of the verification cycle. It proposes a technique to augment the generation of pure-random stimulus with high-value stimulus identified using common patterns and information extracted from a constraint model. Challenges of Pure-Random Generation: In order to understand why it would be desirable to augment pure-random generation during the random regressions phase of the verification cycle, it is necessary to explore some of the downsides of pure-random generation. In our example three-phase verification cycle, both the bring-up phase and the coverage-closure phase provide good metrics on what is being tested relative to defined goals. By contrast, in the random regressions phase, the only meaningful metric of verification progress is bugs found. When a bug is found, the stimulus generated during that simulation is identified by the random seed. This is a good thing in the sense that it allows the stimulus set to be reproduced such that the bug can be examined and corrected. The drawback is that changes to the design and testbench environment – perhaps to resolve the just-discovered defect – change the meaning of the seed and make it impossible to reproduce the same stimulus pattern that uncovered the issue. In addition, when a defect is discovered, ideally it would be desirable to generate new, similar stimulus to the one that uncovered the defect. The random seed provides no information to enable this to be done. Our verification goal during the random regressions phase of the verification cycle is to get to as many corners of the stimulus state space as possible. Random-resistant corner cases are an unfortunate artifact of even the best constraint solver and present an obstacle to achieving this verification goal. Random-resistant cases result from the fact that random stimulus generation is all about probability. A well implemented random constraint solver will generate an even distribution of values across the domain of random variables in the absence of constraints. Take, for example, the simple SystemVerilog class with two random variables shown in Figure 1. View & Download: Read the entire Tackling Random Blind Spots with Strategy-Driven Stimulus Generation technical paper. Source: DVCon 2014