#6 Moore's Law for AI

Will AI Advances Accelerate with Moore's Law? [A ChartPack]

Jul 16, 2023

Welcome to AI Untangled! In this issue we’ll be exploring the growth of AI and investigating the various methods used to track its progress. If you’ve wondered why there has been so much progress in AI recently, the charts in this ChartPack will provide some clarity. As always, your thoughts and questions are welcome!

In This Issue

Introduction
AI Growth
Funding
Training
Patents
Model Size
Publications
Summary

Introduction

Get ready for something new, a ChartPack. In this edition we will try and impress upon you the astounding speed of progress in AI. We will do this using a series of charts. If, like some, your eyes glaze over when you see too many charts, we are here to help.

The charts below illustrate a series of ways to look at progress in AI. We measure a number of different aspects. Looking at these charts, we see a surprising theme emerge across all measures: exponential growth.

This is something you may already be familiar with in the form of Moore’s law. Moore’s law is the prediction that the number of transistors on a microchip would double every two years. This prediction, accurate for 50 years, has driven innovation and progress in the tech industry. The exponential increases in computing power have driven economic growth, transformed entire industries, and improved the lives of billions of people around the world.

In summary, Moore’s law predicts that technology will continue to become smaller, faster, and more powerful, leading to a rapid expansion of technology in every area of life.

The exponential increases in computing power have driven economic growth, transformed entire industries, and improved the lives of billions of people around the world.

How does Moore’s law relate to the progress we are seeing in AI? Two words: exponential growth.

AI Growth

The charts below each describe a different aspect of progress in AI. Each demonstrates an exponential increase. Taken together, we see accelerating progress on many fronts.

Note to our readers: we are emphasizing exponential growth because we, as humans, seem to underestimate it.1 As you read ahead, try and take this bias into account. Whatever you project based on these graphs will likely fall short of what will come to be.

Now, let’s take a look...

Funding

Below is a chart of Annual global corporate investment in AI. This looks like steady progress, but it’s important to note, this is a logarithmic scale. Take a close look at the Y-axis, the increases get larger and larger. Increasing global investment is the fuel of growth in this sector.

Model Training

Training an LLM involves using a large dataset of text to teach a machine learning model how to understand and generate human language. This process requires immense amounts of computing resources; the more compute used, the more capable the models become.2

The graph below is a logarithmic graph of the computation used to train models. The y-axis is measured in petaFLOP; 10^15 calculations. (10 followed by 15 zeroes)

GPT-4 was trained on 10 billion petaFLOP. These numbers are inconceivably large. But let’s try an example: if each of us, roughly all 8 billion of us, performed a calculation per second, non-stop, it would take humanity 400 million years to complete 10 billion petaFLOP of calculations.

Logarithmic Plot Showing a Decade of Progress in Training

Patents

AI Patent filings are doubling every year. This is reflective of the huge increase in effort that the scientific and engineering communities have directed toward AI.

Model Size

Model sizes are essentially the size of the “brain” of the AI. They have been correlated to model capability and continue to grow exponentially.

Source: AWS Presentation “Train and deploy large language models on Amazon SageMaker.”

Why is this important? These models are based on neural networks very roughly modeled after our own brains. The model sizes are quickly approaching the size of our brains.

In this paper published in the Frontiers of Human Neuroscience, the authors argue that the human brain is little more than a “linearly scaled-up primate brain.”

The punchline: our brains seem special compared with our primate friends, but they are not. They are simply larger. The AI models are like primates today, but the models are growing and may soon surpass us in size and capability.

Share AI Untangled

Publications

This graph presents the number of published papers on machine learning and artificial intelligence by year. Each new publication represents new learnings in AI and ML.

This graph was published in Cornell Universities’ arXiv. Link here.

Summary

Exponential growth is difficult to fully comprehend. Humans have a natural bias towards underestimating it. However, with the release of the movie "Oppenheimer" this month, we are reminded of a clear parallel in weaponry. This is not a vague idea of accelerating growth, but rather a visceral demonstration of it in the real world.

Explosive weapons have been built for centuries. But, in 1945, we constructed a weapon that was exponentially larger. "Little Boy" was the codename for the atomic bomb dropped on the Japanese city of Hiroshima by the United States during World War II. It was deployed on August 6, 1945, from a B-29 Superfortress bomber named Enola Gay.

The difference in capability between conventional weapons and nuclear weapons is precisely the type of difference we can anticipate between today's AI and that of tomorrow.