Emerging Narrow Precision Data Types for Embedded AI

Back in 2012, CNNs surged ahead of digital signal processing solutions to become the defacto standard for vision processing tasks like characterization and object detection. The starting point for training and inference of CNN algorithms was Floating-Point 32 (FP32) data types. But it did not take long for inference engines to find ways to optimize the power and area of CNN engines. This was especially important for applications targeting edge device applications. With minimal loss in accuracy, Integer 8 (INT8) became the standard format for CNN algorithms for high throughput use cases. TensorFlow – the dominant AI framework at the time – provided robust support for INT8, although using the data type required post-training quantization and calibration.

When transformer neural networks emerged in 2017 (when Google released their “All You Need is Attention” paper), the addition of attention mechanisms made transformers more sensitive to INT8 quantization than CNNs used for image classification. FP16 and Brain Float (BF) 16 became common fallback data types for transformers.

Transformers enabled the current era of GenAI models, but GenAI models’ parameter sizes are several orders of magnitude larger than CNNs and many Vision Transformers. A typical CNN algorithm might require 25M parameters while ChatGPT requires 175B parameters. This has created a mismatch in computing versus memory bandwidth requirements for an NPU. As you can see in Figure 1, the performance of GPUs targeting AI neural network workloads is growing much faster than the interconnect bandwidth capabilities. 

In 2023, the OCP Microscaling Formats (MX) specification was published and introduced three floating point formats and one integer format (MXFP8, NXFP6, MXFP4, MXINT8). The MXFP8 format is adopted from the OCP 8-bit Floating Point specification (OFP8). Figure 2 shows these data types.

The four MX compliant data types in Figure 2 all have an 8-bit exponent that gets shared across a block of 32 numbers which helps reduce the memory footprint and improve hardware performance and efficiency, which in turn reduces overhead and operational costs. Another benefit of the MX data types is that FP32 or FP16 weights and activations can be “direct cast” (compressed/quantized) into these floating point MX formats during offline compilations. 

As GenAI models continue to evolve, they should follow the same trajectory that CNN models followed. Models will surge in parameter sizes until a satisfactory level of accuracy and efficiency is gained, and then research will pivot to optimizations, making the models more suitable for edge device applications. Both the enhanced Synopsys ARC NPX6 NPU IP and Synopsys ARC VPX DSP IP are available now for SoC designers who are interested in AI capabilities, including GenAI.