Tim Sweeney: Fortnite, Unreal Engine, and the Future of Gaming | Lex Fridman Podcast #467

- Humans are by far the hardest part of computer graphics

because millions of years of evolution

have given us dedicated brain systems

to detect patterns and faces and infer emotions and intent

because cavemen had to,

when they see a stranger determine whether they were

likely friendly or they might be trying to kill them.

And so people in the world have extraordinarily

detailed expectations of a face

and we can notice imperfections,

especially perfect arising

from computer graphics limitations.

Okay, one part is capturing humans.

And so it involved really advanced, dedicated hardware

that puts a human in a capture sphere

with dozens of cameras in them, taking high resolution,

high frame rate video of them as they go

through a range of motions.

And then capturing the human face is complicated

because the nuance detail of our faces

and how all the muscles and sinews

and fat work together to give us different expressions.

So it's not only about the shape of a person's face,

but it's also about the entire range of motion

that they might go through.

So that's the data problem.

There's a lot of other problems with computer graphics.

You know, there's technology for rendering hair,

which is really hard 'cause you can't render every, again,

we know the laws of physics.

It would be easy to just render every hair.

It would just be a billion times too slow.

So you need approximations that capture the net effect

of hair on rendering and on pixels

without calculating every single interaction

of every light with every strand of hair.

That's one part of it.

There's detailed features for different parts of faces.

There's subsurface scattering

because we think of humans as opaque.

But really our skin is, we light travels through it.

It's not completely opaque.

And the way in which light travels

through skin has a huge impact on our appearance.

And this is why there's no way you can paint a mannequin

to look realistic for a human.

You know, it's just a solid surface

and we'll never have the sort of detail you see.

- That kind of blew my mind, like thinking through that.

I think I heard that sort of the oiliness of the skin

creates very specific, nuanced, complex reflections

and then some light is absorbed

and travels through the skin

and that creates textures that are humanized,

able to perceive and it creates the thing

that we consider human, whatever that is.

All of that, while considering all the muscles involved

in making the nuance expression, just the subtle squinting

of the eyes or the subtle formation of a smile,

it's a subtlety of human faces that you have to capture.

Like the difference between a real smile and a fake smile.

But the way to show like beginning of a formation of a smile

that actually reveals a deep sadness.

All of that, like when I watch a human face,

I can like read that.

I could see that.

You have to have the tools

that in real time can render something like that.

And that's incredibly difficult.

- That's right, getting faces right requires the interplay

of literally dozens of different systems

and aspects of computer graphics.

And if any one of them is wrong,

your eye is completely drawn to that

and you find it on the wrong side of uncanny valley.

- The following is a conversation with Tim Sweeney,

a legendary video game programmer,

Founder and CEO of Epic Games

that created many incredible games of technologies,

including the Unreal Engine and "Fortnite,"

which both revolutionized the video game industry

and the experience of playing and creating video games.

This is the Lex Fridman podcast.

To support it, please check out our sponsors

in the description.

And now, dear friends, here's Tim Sweeney.

When did you first fall in love with computers

and maybe with programming?

- I had a brother, Steve Sweeney,

who 16 years older than me,

and at some point when I was a little kid,

he went off to work in California for a tech company

and he'd gotten one of the first IBM PCs.

And so for one summer, I think I was about 11,

I went to visit him in California.

It was my first like trip away from my family

just to hang out with him.

And he had this brand new IBM computer

and I learned to program over the course of a few days

in BASIC, I was just blown away with the capabilities

of computers at the time.

It was unbelievable what they could accomplish.

And I was hooked from that point onward

and very much wanted to be a programmer.

- Do you remember what you wrote in BASIC?

Is it a video game type thing?

Is it like for loop, some numerical thing?

Do you remember?

- Yeah, it's funny, I have a perfectly vivid memory

of all of the first things I learned to program. (laughs)

I have a hard time remembering people's names,

but like code really sticks with me.

Every step and every challenge there were lessons learned

and you know, some of which I've come to realize

were just like me getting over some learning hurdles.

But other things were actually shortcomings

of programming languages

and the realization that there are actually better ways than

what a programmer is learning to program for the first time.

You know, a lot of what they're facing

isn't the challenge of learning a new art.

It's friction introduced by failures

of programming language design.

And so I've constantly come back to those early lessons

there as I've progressed and done more

and more things including building programming languages.

- Yeah, the friction and the pain is the guide

to learning in programming.

Like if I were to describe programming journey

that would be marked by pain.

And that pain, you shouldn't escape the pain.

The pain is instructive for you

to understand programming languages.

But do you remember what kind of stuff you were writing

at that time?

Just the early programs?

- Yeah, in the early days

I wrote a little bit of everything.

I wrote some games.

The first game I wrote on the Apple II was...

Since I only knew how to program in text mode,

the computer would throw asterisks across the screen.

They'd flow from left to right

and you'd have a parentheses on the right hand side

of the screen and yeah, looks like a baseball mitt

and you're supposed to catch the asterisks.

That was my very first game.

It took about a couple hours to build

and tune and I went from there.

But I built a lot of things.

I built databases at different points.

I built a programming language

and a full compiler for a language like Pascal

'Cause I didn't know where you went to buy one of those.

So I made my own.

And you know, one of the fun things at that time

was Bulletin Boards.

Before we had the internet in the hands of consumers,

you used your modem and you dialed into a local phone number

and connected to whoever was running the computer there.

And every town or city

had hundreds of these Bulletin Boards

run by different people

with their own personalities and teams.

And so I spent a lot of time Bulletin Board program

and learning how to deal

with database management and user interface

and dealing with multiple users concurrently and things.

And so I don't know, I'd probably spend about 10,000

or 15,000 hours writing code just on my own as a kid

between like age 10 and you know, age 20

before I actually shipped a program to the outside world.

- 10,000 to 15,000 hours.

What was the value of the hours as a kid you put in,

in programming that led

to the success you've had in later life?

Maybe this is by way of advice to younger people

in terms of how they allocate the hours of their early life.

- Yeah, you know, it's not just hours.

It's really striving to learn to understand

what knowledge you have, what knowledge you lack,

and to continually do experiments

and work on projects that improve your knowledge base.

And I didn't do this with a great amount

of structure or planning.

I was rather just going from project to project,

doing things that I thought would be fun and cool.

And with each project I learned new things.

You know, learning about how to store and manage data,

learning how to deal with advanced data structures,

how to write complex programs

that have deeply nested data and control flow.

Each one of those, you know, provided a lesson

which were later essential, you know,

when in 1991 I released my first game

and over the course of that decade we went from,

you know, zero commercial releases

to the first generation Unreal Engine.

But you know, this was largely just using the knowledge

that I'd built up over the previous decade