In the nightly regression we do use random seeds for running one or several occurrences of our tests in order to hunt bugs that were not expected (scenario not captured in the functional coverage).
Is it a good idea?
In reply to sylvainb:
Yes, you should use random seeds to hunt bugs for unforeseen cases. When you find them, you need to figure out why it took so long to generate the case, and make sure you have the correct functional coverage to make sure it’s tested in the future.
Thanks for your answer Dave
That’s fine, I agree, the following question however pops up, what 's the good amount of bug hunting versus the main goal of the nightly regression (reaching coverage goals)?
The regression is run nightly so that the bug hunting effort can be spread over time.
The drawback I see as well is that using this random seed mechanism in the regression, one may be tempted to delegate the debug of a randomized test, not taking the time to make it robust enough by checking locally many seeds on his test before posting it to the regression…
In reply to sylvainb:
Adding to what Dave stated, you want to ensure that your regressions are optimized so that you are achieving your coverage goals with the minimum number of tests. Most simulation tools will provide you with the ability to rank your tests to determine their contribution towards your coverage goals. You want to re-rank your tests after each regression so that your are being as efficient as possible.
In addition to using the most efficient tests, you will want to run a percentage of additional tests with new seeds. These will possibly find coverage gaps, or provide a more efficient seed to achieve the same coverage.
I’m not sure what you mean by ‘bug hunting effort spread over time’. I would want to find all bugs as soon as possible so that they can be eliminated instead of finding one close to tape out.
In reply to cgales:
The regression will stay during the whole duration of the project and will be run nightly. At the end of the project, if I added 3 months before 10 extra tests with random seeds for bug hunting (bug out of the functional space), I would have run 1000 occurrences that what I meant by spreading over time…
The goal of functional coverage + randomization with the use of test ranking is to make sure that I can reach closure in a deterministic way. Bug hunting is not deterministic, I can always do more.
I agree with what you suggest, i.e. try to eliminate the bugs as soon as possible hence I think that bug hunting process with random seeds should be carried out mainly by the verification engineer who wrote the test by running locally many random seeds, and only to a small extend by the nightly regression.
10% addition on the nightly regression dedicated to test running with random seeds seems reasonable (the other 90% being ranked fixed seed for reaching 100% coverage). What is your opinion?
In reply to sylvainb:
I would run as many additional random seeds as your resources allow. My opinion is that any machine/license not being used is wasted and could be used to find a potential bug.