As per the DO-254 standard, the Airborne Electronic Hardware (AEH) needs accurate assurance of device behavior as intended within optimal operating conditions. For DAL A (Design Assurance Level A) Devices, you need to verify 100% functionality of the device and achieve 100% code coverage, including FEC. Code coverage can be managed using a simulation tool such as Questa®, while functional coverage would require a comprehensive Verification Case Document (VCD) that has cases traced to each requirement of the AEH device.
Once the VCD is defined, it is necessary to ensure accuracy of test bench codes and test cases written to achieve intended operation. To ensure the same, a Verification Procedure Document (VPD) is generated, which consists of test procedures and coverage information for each of the test cases.
Once VCD and VPD are defined, a test plan is generated through linking of both documents, which would prove that all test scenarios are covered with 100% coverage of cover groups, checkers, assertions and test cases. Simulation tools like Questa provide support for generating Test Plan documents. By input of UCDB file using vcover command to Questa, the tool provides the information about the overall coverage and what is not covered. A sample of the Test Plan is shown in Figure 1 below:
Figure 1: Test plan XLS sheet example.
Depth of information covered under Test Plan depends on details received from VCD and VPD. Creating a thorough VPD document is a time-consuming and tedious job. In addition to that, it needs to be maintained, with consistent updates into the VPD document so that both sources remain in sync. For example, if any cover groups/assertions are renamed or added, the same must be updated in VPD.
With rising complexities in the AEH designs, such manual processes won't prove productive and hence risk losing time-to-market. This article gives a simple and fast way of creating such time-consuming and tedious documents using automation.
AUTOMATED VPD AND TEST PLAN GENERATION
Different tools accept testplan XML as an input argument, and this XML can be created by simply saving XLS file as a XML, provided we save it in the correct format expected by the tools. Questa from Mentor has an xml2ucdb command for generating detailed testplan coverage reports sourced from the XML files. Following is the example:
xml2ucdb -debug -verbose -format Excel -dofilename path.do -datafields\"Section,Title,Description,Path,Link,Type,Weight,Goal,ReqtVerified\"testplan.xml UCDB_file
In the example above, testplan.xml can be generated by saving an XLS file having the format as shown in Figure 1 above.
An automated Testplan generation requires the following three steps:
- Creating Verification Case Document (VCD)
- Script for extracting testbench and coverage information and to create VPD document
- Script to create testplan XLS
Step 1 is a manual process and one needs to create VCD based on the requirements with the list of all the test scenarios. In steps 2 and 3, the scripts are required to be generated and users can maintain a certain style of the code while coding test bench and tests, so that the required information can be easily extracted through the script. Nowadays, mostly UVM/OVM methodologies are used which makes steps 2 and 3 easier and faster.
STEP 1: CREATING A VERIFICATION CASE DOCUMENT (VCD)
Every requirement has to be verified in all possible scenarios. Therefore test scenario documents are very important in order to achieve 100% verification completeness. Since requirements of every project differ from each other, the test scenarios must be written for each project. An Excel sheet is a preferred format for documenting test scenarios, as it is easier to read the script in further processes. It can also be created in any tool which supports XLS file export capability. The avionics industry uses different tools to keep records of changes and document versions as per DO-254 standards.
As shown in the table below, the main fields of a test scenario document are Test Number, Test Name, and Test Scenarios. More columns can be added as required. This Excel sheet will be an input file for the script (discussed in Step 3), as show in Figure 2. Additional information related to tests can be added as this will be read by the script and will need no manual effort in merging.
Table 1: Sample of test scenario Excel document.
Figure 2: Illustrates an effective and fast testplan generation flow.
STEP 2: SCRIPT TO EXTRACT TESTBENCH AND COVERAGE INFORMATION AND CREATE VPD DOCUMENT
As per the DO-254 standard, an in-depth Verification Procedure Document (VPD) with detailed information is required to be created. Table 2 below shows some examples of items that are present in VPD. The VPD is a a very lengthy document and has to be maintained to reflect changes in code. Creating and maintaining such a document manually, consumes a lot of time and is even prone to errors.
Table 2: Sample of VPD Excel file with coverage information.
A VPD describes testbench, tests and coverage information, a Perl script can be created which can extract all coverage related information from the ucdb file created after running a test case or a regression. Users can maintain a certain style of the code while coding testbench and tests, so that the required information for tests, checkers, drivers, monitors, sequences, etc. can be easily extracted by the script. The coverage related information can be collected from the merged coverage file too. All the simulation tools provide facility for functional coverage extraction and generate the file in specific format. This file contains required information of cover groups, bins, weight, goal, assertions, etc. The Questa tool creates a ucdb file with all the above-mentioned information.
We can create a Perl script, which can take testbench code and a merged coverage file as an input, to extract different tests, testbench components and coverage related information. The output Excel file of this script is used as one of the inputs for the script described in Step 3. Refer to Figure 2 for the flow.
As shown in Table 2 above, the script should be smart enough to create detailed sheets with relevant information mentioned in columns including "Name", "Purpose", "Type" and "Source File". The output excel sheet shall have detailed information about all verification components like driver, monitor, etc. It can also document checkers from monitor components. The script will automatically grab the description of each component mentioned in the "Name" column and puts it in the "Purpose" column. Information for "UCDB Link" and "Type", related information can be mostly extracted from ucdb (coverage output) file. Information for "Source File" can be gathered through testbench code. Under "Trace" column, a trace ID has to be entered manually, which maps the test case of the VCD document with the particular VPD object.
In order to maintain traceability between output file VPD and test scenario document VCD, a unique link ID called "Trace" needs to be generated. This Trace ID can bridge a particular test scenario from one document to its coverage information mentioned in another document. Without adding Trace ID, the script mentioned in the final step will not be able to link scenarios from the first document to information provided in the second document. It is necessary to preserve the Trace ID when running the script again, or else manual effort needs to be put in for tracing scenarios and its output. To do that, the VPD document can be provided as an input to the script.
STEP 3: SCRIPT TO CREATE TESTPLAN XLS
One more Perl script can be created for final execution. This script takes both the sheets VCD and VPD created as an input and writes to a new Excel sheet to create testplan which will be an output of this script. This output Excel sheet will be a merged document of the information being read from the two input files.
Here, the role of the "Trace" column comes into the picture. The script will read test scenarios from the VCD document (created in Step 1) for a particular Trace ID and reads coverage information for the same Trace ID from VPD (created in Step 2) and creates an output Excel sheet containing all the details, as shown in the table below.
Table 3: Sample of test plan Excel auto-generated by script.
This output Excel document will cover in-depth information of each test case and its coverage in a single document, which gives extra value to the verification plan. It serves as a proof of procedure that was followed for completely verifying each test case. Additional information as shown in Figure 1 above, like instance, weight and goal can be easily added through the script to this XLS sheet to create a final version of the XLS sheet required by the tool as input. Now this test plan Excel sheet can be saved as an XML file to convert to XML format and can be used by the simulation tool to generate final reports.
- One-time effort to create scripts that can be used in different verification projects
- Decreases human efforts
- Saves time and improves productivity
- Less errors while creating VPD documents
- Easy to maintain
- Trace ID has to be filled correctly and manually
- Testbench must be written in specific style to maintain uniformity (works as an advantage too)
Back to Top