Cloud native EDA tools & pre-optimized hardware platforms
Gordon Cooper, Product Marketing Manager, Synopsys
Computing your way from a low-resolution, pixelated image to a smooth, high-resolution output image was an unsolvable problem before deep learning and neural network algorithms. Thanks to breakthroughs in algorithm development, images can be zoomed in 8x or 16x and recreate the resolution. To perform this trick in real-time, you need a processor with a dedicated neural network accelerator, like Synopsys’s DesignWare® ARC® EV processor IP, to handle the intense computational horsepower required without breaking power or area budgets. Image super-resolution techniques are being applied to applications including video surveillance, face recognition, medical diagnosis, digital TVs, gaming, multi-function printers, and remote sensing imaging.
The limitation of all three of these techniques, however, is a theoretic concept called Data Processing Inequality. In simple terms, Data Processing Inequality states that you can’t add information to an image that is not already there. You can’t recover missing data as you zoom into an image. While these techniques can provide a modest amount of sharpening of the original pixelated image, they can’t help you scale up 8 or 16 times.
Or at least that was true until the advent of deep learning and super-resolution neural networks.
Research is now providing variants of SRGANs. Enhanced Super-Resolution Generative Adversarial Network (ESRGAN) has become popular in the gaming community and has been used to upscale vintage video games. Although trained on natural images, ESRGAN applied to a pixelated vintage video came can improve the quality of the graphics.
Another interesting use of super resolution is restoring and colorizing old movies. Super-resolution can scale up the frame rate and resolution, fill in missing data, improve blurriness, and generate realistic colorizations of black and white movies.
The DesignWare EV7x Processor IP family is supported by MetaWare EV (MWEV) software development toolkit, an integrated toolchain for compilation, debugging, optimization, simulation and mapping of the trained neural networks. The MWEV Neural Network Software Development Kit (NN SDK) takes super-resolution graphs like SRCNN, FSRCNN, VDSR and maps then automatically into the EV7x hardware for real-time operation.
Real time implementations vary in performance requirements. A multi-function printer might need to sharpen images at a five fps. However, upscaling from a 30fps video to a 60fps image requires a lot more processing in a shorter amount of time. Requirements impacting processor performance include:
With these parameters, designers can determine which configuration of the deep neural network accelerator they needed. Synopsys can collaborate with designers to provide the fps for different MAC configurations for a chosen super-resolution network based on the input resolution and bandwidth limitations. For example, a multi-function printer implementation might require only the smallest EV71 processor with an 880 MAC accelerator. A high-end gaming application to upscale video on-the-fly might require a larger processor like EV72 or EV74 and up to 3,520 MACs.
In addition to performance, power is often a critical requirement in embedded systems. Power can be calculated once the requirements are known. However, simulating power accurately for neural networks is very difficult given their computational complexity and the time it takes to simulate all those calculations (multiple weeks of time might be needed). Synopsys uses an emulation-based model which is both quick and highly accurate to determine power for the EV7x hardware to implement a neural network.
Bit Resolution Requirements
Neural networks are often trained using 32-bit precision; however, this is overkill for neural network implementation. The EV7x’s DNN provides 8b resolution and up to 12 b resolution when needed for better accuracy. The MWEV NN SDK will quantize the neural network to the chosen bit resolution in hardware. It is also possible to optimize the quantization/precision per layer. Wherever possible, 8b is used and 12b only where accuracy is needed.
Super resolution neural networks will continue to evolve as researchers improve what is today’s state-of-the art. As seen in the evolution on CNN classification networks, the initial focus will be on improving accuracy and then shift to improving algorithm efficiency. The overall goal will be to get the most accuracy with minimum amount of computations and data movement for resizing / upscaling a lower resolution image to a higher resolution image that is pleasing to the human eye. Because the DesignWare ARC EV7x processor IP family is programmable, it can evolve with the constant pace of research into super resolution networks.