There are a number of checks that are performed in a typical UVM environment, some are hidden into a scoreboard, some are provided through assertions, some other checks may laying around in self-checking tests.
When accumulating coverage it is common practice to avoid merging collected coverage when the test fails (although in some cases it might still be useful), so we can be sure the metrics are related to working cases… right? Or is it?
For the sake of the argument let’s make a simple switch example where the path is configured through a configuration register. We build our scoreboard and our model of the switch and then try to send packet across. We collect our coverage through the register bus monitor, we start our regression and we are happy to notice that we hit all possible configuration for the register.
The checks are happening in the scoreboard, which we are delighted to notice did not report an error. Additionally an end_of_simulation_phase() is added to make sure that our scoreboard fifos are all empty, guaranteeing that no “unforeseen” stream of packets has gone through when not intended.
Unfortunately though, one of the members of the packet transaction class was defined as UVM_NO_COMPARE as it was inherited from a previous project where the field was marked as ‘reserved’. This is a serious problem, as the new product is using that field and by looking at our packet coverage we are more than happy to see that everything was correctly covered.
Some may say: “you should have done a review and check that for every coverage item you have there’s a check somewhere”. True, it’s certainly not possible to justify the lack of a thorough review, but is it the only way we could catch the ‘absence’ of a checker for our coverage?
We could potentially enhance the uvm_sequence_item with a bunch of “check_done” flags which are set through a customized do_compare and collect coverage on that meta-data as well, but that doesn’t cover all cases. You still have the issue with self-checking tests where checks are performed (or not) independently from the coverage collection.
For those who had the patience to follow through, here’s the question: how do we make sure that coverage is collected when the relevant checks have been performed?