Bringing Regression Systems into the 21st Century

For years verification management environments within Cypress have been tailored to suit a given project's needs. This led to largely diverging variations of scripts, each of which had to be managed by dedicated resources. Maintaining these multiple management environments quickly became an overwhelming task given increasing complexity and other demands on verification environments. Additionally, having to interface with multiple management environments created a drag on efficiency, making both resources and projects less portable and eliminating opportunities for shared learning and project reuse.

A standard methodology for managing verification was required to provide a uniform interface to users and promote IP reuse. Over time, as enhancements to this standard methodology were made, the entire company benefitted, instead of pockets within the company making improvements.

Ideally, each project can benefit from performance enhancements or efficiency improvements, as well as metrics collection, progress tracking and reporting changes. Furthermore, with consistent metric collection, project regressions can be compared both within and across projects company-wide.

This paper will discuss the verification management infrastructure which includes the following:

• Division of labor between Ruby scripts and Questa Verification Run Manager (VRM)
• Design and testbench tree and file list gathering
• Test tree infrastructure and test lists

This paper will also discuss metrics and report generation which includes:

• Metrics gathering
• Test-level output and reports
• Regression-level output and reports
• Trending of data

Finally, efficient coverage through ranking, including automated seed generation and test ranking with the Questa Verification Management tool, will be discussed.

Verification Management System Infrastructure

Division of Labor

Cypress Verification Management System (VMS) is divided between gathering information about the design and testbench through Ruby scripts, and launching tasks through Questa VRM (Verification Run Manager). While Questa VRM was used for launching jobs, it was not required. A suitable tool could be created to schedule, launch and manage jobs in a server grid. Questa VRM was selected to take advantage of its job-management capabilities as well as its automated features such as coverage merging.

The Ruby script handles all front-end tasks and parses through user created files, including design configuration files; testbench files; and tests, compilation and simulation arguments. These user-created files are named dut.files, tb.files, and vms.cfg and will be discussed in detail later. Configuration information can also be passed through a test list and command line arguments as needed.

Once configuration data is gathered it is compiled and placed into intermediate files to be read by Questa VRM. Questa VRM executes based on a RMDB (Run Management DataBase) file. For the purpose of this project the RMDB was designed to be generic; the database configured Questa VRM to read in two separate files. Both files are a set of TCL arrays representing tasks to be executed by VRM.

The first file contains compilation tasks while the second file contains simulation tasks. Each task consists of TCL array elements providing detailed information about the task such as design or testbench element name, compilation arguments, test name and simulation arguments, among others. The RMDB is primarily an XML file with TCL constructs. This drove the decision to utilize TCL arrays within the intermediate compilation and simulation task files.