by Mark Peryer, Mentor Graphics
THE CRITICAL SUB-SYSTEM
The on-chip bus interconnect has become a critical subsystem of a System On a Chip (SoC). Its function is to route data between different parts of the system at a rate that allows the system to meet its performance goals. The scale and complexity of the interconnect varies from a single switching block, routing transactions between bus masters and bus slaves, to a hierarchy of switches organized to maximize through-put. Verifying that the interconnect works correctly requires that the various bus ports in the system adhere to the protocol specification; that the system address map has been implemented correctly; and that the overall interconnect fabric delivers on its performance goals.
The advent of heterogeneous, multi-processor systems with multiple caches sharing the same data has added significantly to the complexity of interconnect design and has prompted the development of cache coherent protocols such as AMBA® 4 ACETM and AMBA 5 CHITM. Questa SLI supports interconnect verification with a range of capabilities targeting interconnect verification for both simulation and emulation. These capabilities include testbench and instrumentation generation based on Questa VIP; stimulus targeting interconnect and cache coherency verification; and visualization and analysis techniques aimed at giving insight into system level behavior.
THE IMPORTANCE OF AMBA
The process of large scale SoC design integration has been made significantly easier by the use of standard protocols allowing design IP from different vendors to be connected together using defined bus master and slave interfaces. The most prevalent family of on-chip bus protocol standards is the ARM AMBA protocol which is supported by ARM's range of processors, system level IP and the AMBA Designer interconnect generator as well as a large and growing body of proprietary and third party design IP. All of this design IP has to be connected together using interconnect and a disproportionate amount of SoC verification effort is spent on validating that this integration is correct and that the configuration of the interconnect is optimal. As a result, determining the performance of an interconnect is often left until too late into the project to allow it to be optimized. Over time, the increasing scale of integration has meant that SoCs have evolved into complex platforms where several data-hungry sub-systems compete for bandwidth. Being able to analyze the system level behavior to tune the performance has become a must to avoid the area and power consumption costs of overengineering the interconnect fabric sub-system. In order to address these problems, there needs to be a way to reduce the amount of time spent verifying the functionality of the interconnect as well as a way to make it straight-forward to visualize and understand system level behavior. It is particularly important to be able to identify the sources of design and configuration related problems that lead to unsatisfactory performance.
The AMBA protocol family has evolved over time to address the increasing performance requirements of SoCs. The first generation protocols were ASB and APB. The second generation introduced AHBTM which defined a multi-master, multi-slave, multiplexed, interconnect protocol that was limited to in-order read or write transfers with a pipeline depth of 1. The third generation of AMBA protocols introduced AXITM which defined a point to point, master to slave bus interface. Implementing on-chip interconnect for AXI requires the use of a switch to route the AXI master requests to their target slave and then to return the slave response back to the originating master. AXI has become a de-facto standard since it supports high data through-put through the use of multiple sub-channels that separate reads and writes and allow the different phases of the protocol to be pipelined to any depth. AXI also supports out of order responses for read accesses.
The AXI and AHB interfaces of design IP can be verified with Verification IP (VIP) that can check that all aspects of the protocol have been implemented correctly and, for instance, the Questa AXI and AHB verification IP is shipped with a verification plan that can be used to check compliance to the protocol. AXI and AHB based interconnects can be verified using testbenches that use the Questa VIP to inject stimulus that checks the address map and that all supported aspects of the protocol have been implemented correctly. In order to make this process more productive, the Questa SLI product will automatically generate a Questa VIP based UVM testbench for the interconnect using the bus interfaces at its boundary.
Questa SLI generates a testbench environment and configures connectivity graphs according to the design configuration
(stimulus view only, analysis instrumentation omitted).
 |
The generation process then configures connectivity graph stimulus to exhaustively check the address map and those aspects of the protocol that are supported by each bus master. The generation process takes minutes and the graphs typically achieve coverage closure in 10-100x faster than the equivalent constrained-random stimulus. Using Questa VIP the same testbench and stimulus can be used for simulation with Questa and emulation with Veloce.
The other aspect of testbench generation is that it addresses analysis. Transaction information relating to bus transfers and their phases is automatically generated during run time by Questa VIP as protocol signal level activity occurs. In the case of a completely active testbench that wraps around a DUT, the transactions are available from the monitor ports of Questa VIP instantiated in the testbench. The opposite case is a completely passive testbench where Questa VIP is bound to bus nodes which are embedded inside a SoC design which is being stimulated by an alternative verification environment. This type of testbench is referred to as an instrument because it is measuring what is going on inside the device. Questa SLI generates active testbenches, passive instruments or a hybrid of the two when some of the bus interfaces need to be stimulated and others need to be instrumented. The analysis transactions from the testbench/instrumentation are routed to analysis components that process the transactions.
Debugging a functional problem with an on-chip interconnect can be complex especially when there is interconnect hierarchy or there are a large number of outstanding transactions and it is difficult to work out which transactions are associated with each other. In order to alleviate this problem, Questa SLI uses analysis instruments to link transactions together so that it possible to see which downstream transactions are associated with a master request transaction. This linkage is demonstrated in the Questa waveform window – when one of the linked transactions is selected, all of the related transactions are highlighted. This makes it possible to see the 'wood for the trees' when the bus traffic is heavy. The linked transactions are recorded in a database, along with other information, for post-run analysis.
Transaction linking between an AXI Master (M0) and an AXI slave (S3) via an interconnect hierarchy
 |
The other dimension of interconnect verification is establishing that it meets the desired performance. SoCs have evolved into platforms with distributed computing and hardware resources which are capable of being used in any number of combinations. This poses a dilemma for hardware architects since they have to provide enough performance to ensure that most use case scenarios will work while minimizing the area and power of the chip.
There are various ways to approach this, but ultimately they rely on being able to run realistic scenarios that combine application software with actual RTL and input stimulus for major interfaces. The problem with this is that this software and hardware are not usually available until late into the development cycle when it is quite difficult to make changes to the interconnect. To address this, Questa SLI provides a traffic generation stimulus capability. This allows an interconnect to be surrounded by traffic generators that emulate the behavior of hardware blocks using a high level of abstraction called a traffic profile, which defines that this bus master will transfer this much data in this time window. A traffic profile can be targeted to any bus protocol and multiple traffic profiles can be combined to create a traffic scenario that defines a use case. Using a generated testbench and multiple traffic scenarios, the performance of an interconnect can be validated either stand-alone or in combination with real hardware and software as it becomes available.
Example Questa SLI Analysis Charts
 |
In order to make sense of system level behavior, Questa SLI provides a web client to view a selection of analysis charts that allow transaction databases from different simulations to be analyzed or compared against each other. For AXI interconnects, the charts are primarily aimed at looking at the performance and latency. The charts are configurable, allowing different filters to be applied to the data. Using a combination of charts allows the user to view the same information from different perspectives in order to get a better insight into what is going on. The charts are linked to a system level transaction viewer which shows what was going on when a particular performance bottleneck occurred. When a modification is made to correct an interconnect problem, post-run result databases can be compared to validate that the required improvement has been achieved. Other forms of analysis that can be done for this type of interconnect include configurable system level functional coverage, and stimulus quality metrics. System level function coverage looks at the transactions that each master causes to occur in the system and provides feedback on the areas of the protocol that have been used in the stimulus. This type of functional coverage is not normally possible to do with SystemVerilog covergroups since the transactions that are taking place are scattered around different interfaces in the design. Stimulus quality metric charts can be used to determine information about the stress that the stimulus has put the design under.
THE IMPACT OF CACHE COHERENCY
The fourth generation of AMBA protocols added ACE with support for cache coherence which increases the complexity of interconnect verification. With new generations of ARM processors, keeping the processor fed with instruction data has led to multiple levels of memory hierarchy with L1, L2 and L3 caches becoming common place. The caches are not only important to keep instruction and data fetch latencies low, but also to reduce the amount of power that the System consumes by avoiding off-chip accesses to external memory. In a multi-processor system, the different CPUs share areas of data memory to synchronize processes and access to hardware resources. Examples of this shared data are semaphores and command/message tables. Cache coherency is used to allow these shared areas of data to be resident in the data caches of the processors. This means that cache coherent masters can avoid the penalty of making an external memory access in order to check on the state of a semaphore.
In an SoC where the processors are instruction set compatible at the binary level, it is possible to switch the execution of processes from one processor to another; either to equalize compute load; to accelerate execution on a more powerful processor; or to run a process on a lower power core. ARM's big.LITTLETM architecture exemplifies this approach, where different processes can be run on cores in two or more clusters of high or low power processor cores and swapped between clusters. Cache coherence is used to reduce the overhead of switching a process between cores, since it allows the same instruction code to be shared between multiple caches and newly started cores to run from a warm cache.
From a hardware perspective, cache coherency is achieved by allowing a bus master to access the contents of another master's cache using a dedicated interface, a process referred to as snooping. To support cache coherence the AMBA ACE (AXI Coherency Extensions) protocol extends the AXI bus interface by adding more bus fields to its existing channels to support coherent accesses and adding three additional subchannels to support snoop accesses. During an access to shared memory, a coherent master first checks the contents of its own cache, if a cache miss occurs it issues a snoop request transaction to the interconnect. The interconnect is responsible for generating snoop accesses to each of the other coherent masters in the system. If a snoop hit occurs, then the interconnect routes the data from the successful snoop access back to the requesting coherent master to add to its own cache. If all the snoop accesses fail, the interconnect initiates an access to the next level of the memory hierarchy (either a L3 cache or main memory) and returns the result to the initiating master which stores the result in its cache so that it could, in turn, be snooped by another coherent master. For highest performance, an optional speculative memory access can be started in parallel to the snoop. Bus accesses over the ACE protocol do not need to be coherent, and the underlying transport layer is AXI.
There is a variant of the ACE protocol which is referred to as ACELite. An ACELite master is IO-coherent, this means that it is able to access the content of shared caches but that it either does not have its own cache or does not share its contents with other masters. At the interface level, ACELite contains the ACE field extensions to AXI, but it does not include the three extra snoop channels. Examples of ACELite masters include GPUs or other hardware accelerators that would pick up a command/message table from an area of shared memory.
Transaction linking in an ACE system, Master_0 issues a request for data from shared memory, the interconnect issues snoop transactions to the other coherent masters in the system, if these fail the interconnect accesses external memory via Slave_2.
 |
The ACE protocol relies on snooping all shared coherent masters in the system, although this is more efficient than going to main memory for each access, there is a finite limit to the number of coherent masters that can be supported in such a system. The latest AMBA 5 CHI (Coherent Hub Interface) protocol, announced in June 2013, is designed to address this problem and can be scaled to support a large number of coherent masters. The CHI (Coherent Hub Interconnect) AMBA-5 protocol, announced in June 2013, is designed to address this problem and can be scaled to support a large number of coherent masters. It takes a different approach to snooping from ACE, using a cache directory to ensure that a coherent master is only snooped if it is likely to have the required cache line. CHI also breaks away from the AXI interface to use a high speed packet oriented approach for the underlying protocol transport layer. Like ACE, CHI supports both coherent and non-coherent traffic.
Questa SLI instrumentation links ACE and CHI transactions together so that it is possible to see which master request caused which snoop transactions, and in the case of a snoop miss, the associated access to main memory. Adding the snoop channel means that checking a cache coherent ACE or CHI master interface is no longer limited to a one-to-one master-slave relationship, since the system level complication of the snoop interface also needs to be taken into account.
The snoop channel is not the only additional complexity. With a non-coherent cache each cache line has a status bit that indicates whether the data in the line is valid or not. This enables the cache controller to determine whether the result of a cache access is a hit or a miss. With coherent caches, more states are required to keep track of the state of the data. The ACE protocol defines 5 cache states—Invalid (I), Unique Clean (UC), Unique Dirty (UD), Shared Clean (SC) and Shared Dirty (SD). The Invalid state means that there is no valid data stored in a cache line. If the cache line state is unique, then it means that the data only exists on that master and nowhere else in the system, whereas if it is shared, then the data exists in at least one other coherent cache. If the data is clean, then it means that its content has not changed since it was fetched from main memory, but if it is dirty then it has been updated by a write and the version in main memory is out of date. The state stored in each cache line represents one element of a volatile FSM which is distributed across the equivalent cache lines in other masters. In other words, the transitions between the cache states are affected by the current cache line state and the state of equivalent cache lines that may or may not exist in other coherent caches. In addition, the ACE protocol supports 17 different types of snoop transactions which cause different transitions to occur in the cache line state according to whether the cache is being snooped or whether the cache is being updated after a snoop hit. In principle, verifying that a coherent cache controller has made all of the possible state transitions for all combinations of snoop transactions and response conditions is pretty much unachievable for all the cache lines. However, if the verification process is focused on a few cache lines, then the overall behavior of a cache controller can be validated by the principle of reduction.
Distributed ACE Coherent Cache State Machine – Each cache line can be in any of these states, and transitions between states rely on a combination of local and remote states and the type of transaction taking place.
 |
Even when working with just a few cache lines it is difficult to provoke all of the possible cache line state transitions. Techniques such as constrained-random stimulus are ineffective when faced with this challenge, which involves co-ordinating accesses between masters to create the desired state transitions. Questa SLI provides a combination of directed and graph-based stimulus which is designed to target the validation of cache coherent systems. Questa SLI provides post-simulation coverage analysis charts for cache line transitions. This is available for a single test case database, or a virtual merge of the results of several test cases. Since the state transitions from transactions received via the snoop channel are different from those generated by the request channel, there are two cache line state transition charts. These allow users to determine the quality of their stimulus.
Other charts provided by the Questa SLI client allow other aspects of the cache coherency stimulus quality to be determined. For instance, it is possible to look at the number of outstanding transactions that are queued at each master and slave either as a distribution or at a particular point in time, to give an indication of the degree of stress that the stimulus has placed on the system. Another quality metric is the 'distance' between transactions in terms of time and address locality. Again, this gives an indication of what stress the coherent masters have been subjected to, and what spread there has been in the stimulus applied. In addition to the normal transport level bandwidth and latency performance questions, the performance of a cache-coherent system is affected by the cache hit rate and the snoop hit rate. It is also affected by the snoop latency, in other words how long a snooped master takes to respond to a snoop hit. Questa SLI provides charts that allow these metrics to be analyzed, and these charts are linked to the transaction viewer so that root causes of problems can be identified.
Questa SLI Cache Coherency Phase Level Latency and Cache Line State Transition Charts
 |
IN CONCLUSION
Successful verification of the on-chip interconnect is critical to the successful completion of an SoC. Questa SLI builds on top of Questa bus protocol VIP to provide a powerful combination of code generation, testbench infrastructure, configurable stimulus and analysis capabilities that make it possible to functionally verify an interconnect fabric and then to check its performance under different loads. When a throughput issue is found, it can be traced back to its root cause, and the analysis infrastructure allows the behavior of different versions of the implementation to be compared.
In addition to support of non-cache-coherent protocols such as AXI, Questa SLI fully supports the ACE and CHI cache-coherent protocols. The combination of its powerful stimulus generation and visualization techniques make sure that cache coherency verification is thorough and effective.
Support of the AMBA protocol family
| Protocol |
ARM Fabric |
Questa VIP Support |
Questa SLI Generation |
Connectivity Graph |
Traffic Generation |
Analysis Support |
|---|
| AHB |
NIC301/NIC401 |
X |
X |
X |
X |
Performance, Stimulus Quality Metrics |
| AXI3 |
NIC301/NIC401 |
X |
X |
X |
X |
Performance, Stimulus Quality Metrics |
| AXI4 |
NIC401 |
X |
X |
X |
X |
Performance, Stimulus Quality Metrics |
| ACE |
CCI400 |
X |
X |
X |
X |
Performance, Coherency coverage & stimulus quality Analysis |
| CHI |
CCN504 |
X |
X |
X |
X |
Performance, Coherency coverage & stimulus quality Analysis |
REFERENCES:
- AMBA AXI and ACE Protocol Specification—ARM 2013 - ARM IHIH 0022E
- Traffic Profiling and Performance Instrumentation For On-Chip Interconnects—Mark Peryer and Bruce Mathewson—DVCon 2013
- Verifying AMBA Interconnect Performance Using Database Analysis and Traffic Generation—Mark Peryer and Gordon Allan—ARM TechCon 2013
- SQLoC: Using SQL database for performance analysis of an ARM v8 SoC—Avidan Efody and Gordon Allan—ARM TechCon 2013
- Verifying an AXI Fabric Using a Graph Based Approach—Galen Blake and Mark Olen—ARM TechCon 2013
Back to Top