OpenAI is an AI research laboratory. Its rather far-reaching mission is to ensure that artificial general intelligence benefits all of humanity. The lab also studies and publishes information about the way AI is growing and how it’s being used. If you have followed Moore’s law over the years, some of these facts will get your attention.
OpenAI points out that since around 2012, the amount of compute used in the largest AI training runs has been increasing exponentially, with a 3.4-month doubling time. Remember Moore’s law had a roughly two-year doubling period. This means that since 2012, the increase is in excess of 300,000x. Note that a two-year doubling would yield a 7x increase. In my view, this defines explosive growth. In the words of Stelios, “Software is eating the world, and AI is eating software.” Let’s look at what this means with a specific example – a program that learns to master the game of Go, which is significantly more complex than chess. For perspective, chess has about 10123 states and Go has about 10360 states.
According to DeepMind, AlphaGoZero uses a novel form of reinforcement learning in which AlphaGoZero becomes its own teacher. If the algorithm was able to compute at a rate of one petaflop, the process would take about three years. Note one petaflop is equal to one thousand million million (1015) floating-point operations per second, so this isn’t slow, just not fast enough. The throughput of new AI hardware is mind-boggling.
Some notable quotes are worth repeating here. This will provide a broader perspective on what’s ahead:
- “NOR flash enables 50x denser weight storage, resetting Moore’s Law.” - Michael B. Henry, Mythic
- “46,255mm2 of silicon, 400,000 cores, 9 PByte/s memory bandwidth – training on 45,000 years of human intelligence.” - Andrew Feldman, Cerebras
- “Data center power consumption doubling every year (1/5 of all energy produced by 2022).” - Zaid Khan, Qualcomm
That last one should give you pause. We have a long, long way to go regarding energy efficiency. Low-latency speech recognition can demand the equivalent energy consumption of ~41 U.S. households. By contrast, the human brain is two to three orders of magnitude more efficient than today’s silicon-equivalent processing power, delivering 1016 FLOPS with about 20 watts of power consumption. And all this in the face of component variability with multiple failure modes. So, be excited, but realistic about what is left to conquer.