What Apple’s Neural Engine Tells You About the Next Decade

Okay, I just got the new MacBook Pro and

I want to talk about it, but not in the

way that we typically hear people speak

about it. We're not going to go through

the general specs, the things that you

can easily search up online. We all know

about that. I want to talk about a

bigger story here. If you sit down and

read about the reviews, you hear about

the CPU, you'll hear about the different

specs because those are the numbers

people know how to compare. I mean,

Geekbench scores, render times, frame

rates. But there's also a 16 core neural

engine inside this machine that barely

gets mentioned. And Apple has been

making it bigger every single generation

since 2017. And so have other companies.

I mean Qualcomm, Intel, really every

major chip company on Earth is quietly

dedicating more and more transistors to

the same type of component. And when

that happens, when companies that maybe

aren't aligned on other things, but they

all start making the same thing or the

same investment at the same time, that's

not a product feature. That's a signal.

And if you know how to read that, you

can see exactly what they think is

coming. And for this video, we're going

to focus on Apple. So, I'm very inspired

by my new MacBook. But first, I want to

say a big shout out to Surf Shark, who

sponsored this video. I mean, I travel

constantly for work. GTC, CES, I mean,

you name it. Every single time I am

sitting in an airport or hotel lobby

connected to public Wi-Fi, sending

files, I mean doing so many things,

things I should probably not be doing on

an open network, but I need to. And that

is where Surf Shark comes in. It

encrypts everything between your device

and the internet. So nobody on the

network can see what you are doing. And

the other thing I use it for all the

time is changing my virtual location.

Surf Shark has over 4,500 servers in 140

locations. So, if I'm traveling and

something is georestricted, I can just

connect to a server back home and I'm

good. Another thing I really love is one

account works on unlimited devices, too,

which is really unheard of. All right,

so if you want to go sign up, go to

surfshark.com/tiffech

or use the code tiffant at checkout to

get four extra months free. The link is

in my description. And there actually is

a 30-day money back guarantee. So,

there's really no risk. It's just

preventing risk, especially when you're

using public Wi-Fi. All right, so we

have first we have to step back a sec.

Apple left Intel back in 2020. That's

when they shipped the M1, their first

custom silicon for Mac. And at the time,

a lot of people treated it like a

gamble. Could Apple really design a

laptop chip as good as say Intel's? Now,

well, 6 years later, the answer is

pretty clear. Absolutely. But the more

interesting thing is how they did it.

Because Apple's approach to chip design

is fundamentally different than everyone

else in the industry. Now let's go

through an example of this. Intel, we'll

use them as an example, designs a

processor. Then Dell puts it in a

laptop. HP puts it in a different

laptop. Another company puts it in

another one. So this one company

controls the chip. The manufacturer

controls everything else. So many other

companies work the same way. They design

the silicon, license it out, and then

other companies build products around

it. And it works. It's a great way to do

things. But Apple, they design the chip.

Apple designs the memory architecture.

Apple writes the operating system. Apple

builds the developer frameworks that

software is written against. It's one

company, one entire stack from the

transistor all the way to the API. And

that has really interesting things that

you can actually see. I mean, take

unified memory. On a traditional PC, the

CPU has its own memory and the GPU has

its own memory. And when the data needs

to move between them, it gets copied.

That takes time and power. On the M5,

the CPU and GPU share one pool of

memory. So there's no copying. The data

is just there. And Apple could do that

because the team designing the chips is

the same team designing the operating

systems memory scheduler. They built

them together. Now let's go back to the

neural engine. This is something that is

so underrated in my opinion. I mean this

benefits from the same integration. It

sits on the same die accessing the same

memory pool and the same software that

runs on it which is a framework called

core ML and it was built by people who

sit down the hall from people designing

the transistor layout. I mean I don't

know if it's right down the hall but you

get what I'm saying. When you control

the full vertical like that every layer

can be optimized for layers above and

below it. And this in my opinion is

where it gets really interesting. The M5

introduced something new. For the first

time in an M series chip, Apple embedded

neural accelerators directly inside the

GPU cores. So, the neural engine is

doing its thing on one part of the chip

and now the GPU can also run AI

workloads natively. And Apple claims it

has over four times the peak GPU compute

for AI compared to the M4. Now, that's

the kind of design decision that only

happens when one company owns the entire

pipeline. You can coordinate what the

neural engine handles versus what the

GPU handles because you control the

software routing those workloads. And

really nobody else in the PC space can

move like that. And then the M5 is

generation 5 of this approach. So same

architecture scales from a 599 MacBook

Neo all the way up to Mac Studio. I mean

5 years ago this silicon program didn't

exist. Now it's powering every Mac Apple

sells.

Okay. Okay, so when I was doing research

on this, this is where, in my opinion,

the story gets pretty fun. I want to

show you something. In 2017, Apple put

the first neural engine in the A11 chip,

the one inside the iPhone X. It had two

cores. It could do 600 billion

operations per second. And that sounds

like a lot, but all it really did was

power Face ID and Animoji, and

developers couldn't even access it.

Apple kept it locked through their own

software. Now, one year later, A12

jumped to eight cores. 5 trillion

operations per second, nine times

faster, using a tenth of the power. And

this happened, or this time, Apple

opened it up to everyone. They released

a framework called Core ML that

essentially let any developer run

machine learning models on the neural

engine. Fast forward to 2020, the A14

doubled the core count to 16 and hit 11

trillion operations per second. That

same year, Apple shipped the M1 with an

identical neural engine, bringing it to

the Mac for the very first time. Then

the M4 pushed to 38 trillion operations

per second. And now the M5 takes that

further with a faster 16 core neural

engine, plus those new neural

accelerators in the GPU. Stay with me

for a sec here. Follow that arc for a

sec. Two cores to 16, 600 billion to 38

trillion, 8 years every single

generation. And Apple chose to dedicate

more transistors to this component. And

transistors are expensive. Chip real

estate is finite. Every square meter you

give to the neural engine is a square

millimeter even you don't give to say

the GPU or the CPU. And Apple kept

making that trade-off anyways. And

they're not alone in doing that. There

have been a few other companies that

have done the same thing or something

similar. Anyways, for example, Qualcomm,

their new X2 Elite coming this year

pushes that to 80 trillion. So when you

zoom out and see that pattern across

different companies, you're watching

something form in real time. So everyone

agrees that AI compute matters. The