Finding Hardware Bugs Faster with Full Visibility Debug

The ever increasing acceptance and adoption of physical prototyping by IP and SoC developers into wider ranging validation cycles is the direct result of the evolving implementation and debug automation technologies that have rapidly extended beyond must-have partitioning, pin multiplexing and implementation flows into advanced debug and signal visibility flows. 

The significantly eased bring-up and increased debug capabilities have driven the adoption of physical (FPGA-based) prototyping for even earlier IP and SoC validation phases. In the past the creation of physical prototypes was very labor intensive and debug visibility was very limited. As a consequence, prototypes were placed late in the design cycle after RTL was very stable in order to serve software developers for pre-silicon software bring up and validation. Today, through integrated prototyping solutions and automation by HAPS ProtoCompiler, bring-up effort has been reduced from month to less than two weeks. RTL changes no longer result in manual re-partitioning, routing and insertion of time domain multiplex IP, which is all automated, resulting in increments in a matter of days. In addition, HAPS’ debug capabilities have been greatly evolved. This along with HAPS’ unique performance makes it an attractive vehicle to spot the hardest to find bugs in the design which may only appear after running.

The ability to monitor many multiple thousands of signals per FPGA without impacting the system speed has become possible through the introduction of the HAPS Deep Trace Debug (DTD) technology in conjunction with the HAPS ProtoCompiler implementation and debug tools. This topic was covered in detail in a previous article by Achim Nohl in an earlier edition of this newsletter “Preparing Your Prototype for Better Debug”.

HAPS DTD comes in various configurations to suit many different debug requirements, but all allow the user to get valuable signal visibility within each of the FPGAs on the HAPS system along with sample depths extending up to millions of clock cycles and complex triggering scenarios that can be defined during the runtime execution. In a typical use-case, a user might want the debug samples to be downloaded into a text file, a Value Change Dump (VCD) file for viewing in a simulator, or most notably into a Fast Signal Database (FSDB) file that can be read into the Verdi and Siloti debug environment. HAPS physical prototyping in conjunction with Verdi and Siloti provides a very powerful debug environment, offering a number of integration options that can be deployed depending on the specific debug need.

HAPS DTD is now a built-in capability of the latest HAPS-80 physical prototyping systems meaning that anyone using the latest generation HAPS has this debug capability at their immediate disposal without the need for additional debug hardware or software implementation and debug tools. Full implementation automation is enabled via the HAPS ProtoCompiler tools while multiple interactive debug sessions are enabled through the HAPS ProtoCompiler runtime tools. This means that HAPS DTD should be considered as a default implementation option and deployed to provide valuable signal visibility for all physical prototyping implementations as opposed to just an “after the event” debugging tool to assist with suspected platform bring-up or FPGA implementation issues.

Such enhancements to the HAPS DTD debug technologies has resulted in the adoption of physical prototyping for verification tasks beyond traditional software development and real world interface testing where debugging would be enabled via software debuggers and real I/O interfaces respectively and where visibility into the individual FPGAs might not be the most important factor as the hardware RTL is expected to be in an advanced state for these specific verifications tasks.

An example of the extended adoption of HAPS Physical Prototyping has been the recent use for the debugging of IP RTL in order to help trap corner-case bugs that might only happen after extended periods of operation, that might not be possible to reach within available simulation time-slots, and in unexpected scenarios that might not be covered with regular simulation test-benches. In these cases, the IP would be run at high-speeds on the HAPS prototyping systems for periods which may extend to several weeks while the HAPS DTD logic is used to monitor signal or bus activity for any trigger scenarios of interest, which may range from simple signal transitions through complex bus and control signal sequences. The ability to define a wide range of trigger scenarios at runtime, without impacting system performance, and use any combination of the thousands of signals selected as debug points during the initial HAPS prototyping system configuration has in turn seen the recent adoption of physical prototyping farms. These farms consist of large numbers of HAPS systems are configured with common implementations but then run with different firmware builds, software revisions and trigger scenarios to help trap multiple debug conditions concurrently.

Once any of the triggers have fired, the HAPS ProtoCompiler Debugger can then download the sample data from external memory modules on the HAPS prototyping system and write the data to text, VCD or FSDB files for off-line debugging before the HAPS systems can be re-armed and re-started to wait for the next trigger scenario. The FSDB file in particular has become a popular choice for debugging as it’s open format allows for different levels of debug abstraction. One common use for CPU debugging has seen the use of scripts to generate a textual output from the FSDB file which provides the CPU designer with visibility at an Instruction level, from which the designer can decide where to start analyzing the level-level waveforms.

A key advantage HAPS DTD offers by having visibility of many thousand signals is that more debugging is possible from a single prototyping system implementation. It means that fewer iterations might be required in order to complete the debug or validation cycles by reducing the need to switch otherwise limited signal visibility to other areas of the device under test (DUT). With iteration times of high-capacity prototyping platforms still taking many hours, it might mean that limited debugging is possible on a daily basis before needing to re-spin the implementation with revised debug signals sets. In any case, limited signal visibility is one of the reasons physical prototyping has mainly only been deployed when the RTL is in a near stable and where visibility is only required on key buses and signals.

Another interesting situation is when it might have taken many days for a specific complex trigger to “fire” only to discover that visibility wasn’t selected on necessary signals. This is where HAPS Global Signal Visibility (GSV) helps open up new frontiers for physical prototyping by enabling a “snap-shot” view of the full system state.

HAPS GSV is a recent enhancement to the debug capabilities of the HAPS physical prototyping solution that allows the state of registered instances (full system state) to be captured from the DUT without the need for pre-defined debug probe selection. As with HAPS DTD, there are a number of ways in which GSV can be set-up and deployed to help debug different scenarios but perhaps one of the most interesting is through the control the DUT clocks (Synchronous GSV) to hold parts, or even the complete DUT in a static state whilst capturing the registered instance values. In this case, the user can decide which of the DUT clocks they want to control and under what circumstances they want to control, which can be through a command issued from the Host PC or for finer control, by using the trigger from the HAPS DTD logic. Once the full system state has been captured, the controlled clocks can then be released to resume normal platform operation, or stepped by a user defined number of cycles before capturing the full state once again. This same process can then be repeated any number of times, even 10s of thousands of times, to collect the full system state of the DUT over a useful period of time. As with the HAPS DTD flow, the system state can then be downloaded and written into VCD file or FSDB database for further debugging in the chosen off-line debug environment.

Again, the use of FSDB files in the Verdi environment allows for extended debug opportunities and in the case of GSV, the use of the Siloti Data Expansion capability can expand the signal visibility to any combinatorial point between the registered instance values as collected from the HAPS physical prototyping system.

When HAPS GSV was first introduced, the one likely use case was to help diagnose unexpected DUT lock-up conditions, where all or parts of the DUT were stuck in a static state. Instead the ability to finely control DUT clocks from the HAPS DTD trigger and to step it over large number of cycles has seen the most adoption, specifically for verification needs where the capture of all registered instance allows for visibility into areas of the DUT that might not be covered by even the thousands of pre-defined debug signals.

HAPS DTD still plays an important part here as those thousands of debug signals can still be used to help define all the simple and complex trigger scenarios to allow the HAPS system to progress the DUT to the point of interest at full system speed, which again might be many days or even weeks, before the DUT clocks are then stopped and stepped to collect the full system state over a subsequent period of time, but at that point of interest.

One example of such an adoption of HAPS DTD and GSV for earlier verification cycles has been for earlier RTL debug where the system speeds of the HAPS system allows the RTL designer to progress to extended points of interest that might not be possible with simulation but then to get simulator-like visibility at that point of interest. In these cases, the RTL designer might not really know which signals or buses he might want to inspect in order to find the current issues, but the ability to run multiple tests with different trigger scenarios and to browse around all part of the DTD without having to build a new HAPS implementation enables a speedy debug process – visibility into the FPGA is the key need here.

Historically, the deployment of any physical prototyping technology within any IP or SoC development and verification program has been limited by the amount of signal visibility that is possible into the actual DUT for the specific development and verification need. The introduction of the HAPS DTD and GSV technologies, to complement HAPS’ existing debug capabilities, now allows for extended verification opportunities through multiple debug options along with greater signal visibility and sample depths.